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I.  INTRODUCTION 


A.  BACKGROUND 

Military  program/project  managers  (PMs)  are  under  ever-increasing  levels  of 
scrutiny  to  ensure  that  major  acquisition  programs  meet  cost,  schedule,  and  performance 
goals.  As  budgets  within  the  Department  of  Defense  (DoD)  decrease,  PMs  must  take 
much  greater  care  to  optimize  limited  budgets  and  manpower  resources  in  order  to 
provide  the  greatest  value  to  the  taxpayer  and  the  end  user. 

One  of  the  most  important  decisions  PMs  make  is  how  to  optimally  allocate 
resources  among  the  various  upgrade  options  for  a  given  program.  Trade-off  decisions 
made  by  PMs  when  considering  which  upgrades  to  purchase  are  vital  to  ensuring  value 
maximization  during  system  upgrade  projects.  Generally,  trade-off  decisions  involve 
identifying  which  upgrades  are  the  most  critical  as  well  as  the  optimal  amount  of 
upgrades  to  purchase  from  a  range  of  upgrade  options,  determining  how  best  to  allocate 
these  across  the  force,  and  staying  within  the  constraints  of  the  budget. 

In  his  Better  Buying  Power  2.0  memorandum,  Frank  Kendall  (current  under¬ 
secretary  of  defense  for  Acquisition,  Technology,  and  Logistics)  stated  that  delivering 
better  value  to  the  taxpayer  and  warfighter  by  improving  the  way  the  DoD  does  business 
is  his  top  priority  (Kendall,  2012).  Most  applicable  to  this  thesis,  he  mentioned  that  it  is 
possible  that  programs  will  be  halted  if  they  do  not  factor  cost  trade-offs  into  their  efforts 
to  reduce  the  overall  cost  of  the  program.  He  also  stated  that  unless  these  trade-offs  are 
considered,  “the  Department  will  continue  to  spend  billions  on  development  and  initial 
production  of  programs  that  are  ultimately  canceled  or  curtailed”  (Kendall,  2012, 
para.  3).  Optimizing  trade-off  decisions  results  in  deriving  the  best  overall  value  with  the 
resources  available.  This  report  focuses  on  identifying  a  method  to  help  PMs  recognize 
potential  cost  trade-offs. 

The  primary  approach  to  identifying  such  a  method  was  the  application  of  utility 
modeling,  which  is  a  tool  that  can  assist  decision-makers  in  quickly  identifying  optimal 
trade-off  solutions  for  a  given  data  set.  Identifying  and  quantifying  constraints  is  a 
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critical  aspect  of  utility  modeling,  and  the  process  of  identifying  these  constraints  will 
assist  PMs  in  making  good  trade-off  decisions. 

In  order  to  find  an  optimal  trade-off  solution,  individual  upgrade  quality  must  be 
defined  for  any  given  project.  When  using  a  utility  model  to  assist  in  decision-making, 
quality  must  be  quantified.  This  is  often  a  difficult  and  time-consuming  process  because 
there  are  many  factors  that  must  be  weighed  when  determining  a  quantitative  value  for 
quality.  Factors  such  as  cost,  schedule  impact,  improved  capability,  fielding  impact, 
maintainability,  durability,  ergonomics,  and  so  forth  all  make  up  quality  and  these  are 
unique  to  each  program.  Balancing  the  trade-offs  between  cost,  schedule,  and 
performance  embodies  the  art  and  science  of  program  management.  This  research 
project  developed  a  utility  model  based  on  linear  programming  that  can  be  tailored  and 
applied  to  any  DoD  acquisition  program  to  determine  an  optimal  upgrade  allocation 
(Balakrishnan,  Render,  &  Stair,  2011). 

In  conducting  our  research,  the  authors  used  the  Apache  Block  III  (AB3) 
Modernization  Program  as  a  means  to  test  the  accuracy  of  the  utility  model.  The  AB3 
Modernization  Program  is  managed  under  Program  Executive  Office  (PEO)  Aviation  at 
Redstone  Arsenal,  Alabama.  The  AB3  is  an  Apache  attack  helicopter  modified  to 
effectively  and  efficiently  integrate  the  Longbow  Apache  well  into  the  21st  century.  The 
AB3  Modernization  Program  is  a  multi-billion  dollar  upgrade  program  that  involves  a 
wide  range  of  upgrade  options.  The  plentitude  of  upgrade  choices  presents  an  ideal 
testing  ground  for  the  utility  model  developed  in  this  project. 

The  AB3  is  designed  to  provide  a  significantly  enhanced  warfighting  capability 
over  the  AH-64A  and  AH-64D  models.  Some  of  the  improvements  to  the  Apache 
include: 

•  Longbow  fire  control  radar  (FCR) 

•  Modernized  Target  Acquisition  Designation  System/Modemized  Pilot 

Night  Vision  System  (MTADS/M-PNVS) 

•  Longbow  Hellfire  missiles 

•  integrated  command  and  control  (C2) 

•  intelligence/surveillance/reconnaissance  (ISR)  improvements 
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•  communications  connectivity  for  attack/reconnaissance  aviation  within 
brigade  combat  teams,  divisions,  and  corps 

•  improved  engine  perfonnance  and  reliability  (Department  of  Defense, 

2012). 

B.  PURPOSE 

The  purpose  of  our  research  is  to  determine  whether  utility  modeling  based  on 
integer  linear  programming  can  assist  PMs  in  finding  the  optimal  allocation  of  scarce 
budgetary  and  manpower  resources  for  block  upgrades  to  MDAPs.  The  goal  of  our 
research  is  to  develop  a  utility  model  that  can  be  applied  to  any  DoD  acquisition  program 
to  assist  PMs  in  more  effectively  allocating  resources. 

C.  RESEARCH  QUESTION 

Our  primary  question  is:  Can  utility  modeling  be  used  to  effectively  find  an 
optimal  allocation  of  upgrade  purchases  when  choosing  from  a  range  of  potential 
upgrades?  In  this  project,  we  consider  a  successful  trial  as  an  iteration  wherein  decisions 
made  by  PMs  using  the  utility  model  are  at  least  90%  consistent  with  decisions  made 
using  more  methodical  and  time-consuming  approaches.  One  such  process  is  the 
program  objective  memorandum  (POM)  process,  where  a  program’s  team  intensely 
reviews  all  potential  upgrades  and  develops  a  user-agreed  optimal  mix  of  upgrade 
solutions. 

D.  ASSUMPTIONS  AND  LIMITATIONS 

This  research  project  was  written  with  the  assumption  that  the  reader  has  a  basic 
understanding  of  the  defense  acquisition  process  and  the  role  the  PM  has  in  making 
upgrade  decisions  as  they  pertain  to  the  program  that  is  being  managed.  In  this  research, 
we  assume  that  the  PM  has  the  authority  to  decide  how  to  optimally  allocate  budget  and 
manpower  resources  to  get  the  best  overall  value  when  making  upgrade  decisions. 

Due  to  time  and  resource  constraints  in  our  research,  we  only  applied  this  model 
to  the  AB3  Modernization  Program.  In  order  to  refine  the  utility  model  further  and  to 
determine  the  model’s  usefulness,  examining  several  programs  is  ideal.  While 

conducting  our  research,  we  found  that  quality  is  a  difficult  element  to  quantitatively 
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measure  because  it  changes  with  each  program.  In  the  program  we  studied,  getting  a 
complete  grasp  and  quantifying  what  quality  is  was  a  major  factor  in  determining  the 
effectiveness  of  the  utility  model. 

E.  METHODOLOGY 

To  conduct  our  research,  we  used  several  types  of  data  and  interviews  to  focus 
our  efforts.  We  started  with  literature  reviews  of  federal  law,  Office  of  the  Secretary  of 
Defense  (OSD)  guidance,  academic  journals,  modeling  textbooks,  private  and 
government  websites,  federal  acquisition  regulations,  DoD  instructions,  government 
reports,  and  third-party  books.  Reviewing  this  literature  provided  us  with  a  basic 
understanding  of  utility  modeling  and  acquisition  processes  and  helped  us  focus  our 
research  efforts.  We  also  conducted  extensive  interviews  with  the  PM  of  the  AB3 
Modernization  Program. 

When  developing  the  model,  we  created  a  worksheet  to  derive  the  quality  value  of 
all  potential  upgrades  being  considered.  We  divided  this  worksheet  into  multiple 
categories  and  created  a  fonnula  to  generate  a  quality  index.  Next,  we  identified  as  many 
constraints  that  the  PM  has  to  contend  with  and  can  influence  as  possible.  Once  we 
developed  all  these  factors,  we  applied  them  to  the  utility  model  to  determine  the  optimal 
upgrade  mix  for  a  PM  to  allocate  funds  toward. 

F.  ORGANIZATION  OF  REPORT 

The  report  is  structured  in  a  methodical  way:  (a)  we  lay  the  framework  for  the 

model  (the  requirements  for  and  the  theory  behind  the  model),  (b)  we  detail  how  the 

model  was  developed,  (c)  we  apply  the  model,  (d)  we  analyze  the  results,  and  (e)  we 

present  ideas  for  follow-on  research  and  improvements  to  the  model.  In  this  chapter,  we 

detailed  the  background  and  methodology  for  the  report.  In  Chapter  II,  we  summarize 

our  literature  review  to  provide  a  theoretical  basis  for  modeling  and  to  provide  data  on 

the  AB3  Modernization  Program.  In  Chapter  III,  we  show  how  the  model  was 

constructed  and  present  the  upgrade  options  available  to  that  program.  It  is  in  that 

chapter  that  we  detail  the  PM’s  constraints  in  tune  and  costs  and  show  the  quality  ratings 

of  the  potential  upgrades.  In  Chapter  IV,  we  apply  the  model  to  the  AB3  upgrade 
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possibilities  to  determine  the  optimal  amounts  and  types  of  upgrades.  In  Chapter  V,  we 
compare  the  results  of  the  model  to  what  was  actually  determined  to  be  the  optimal 
resource  allocation  by  PM  Apache.  And  in  Chapter  6,  we  make  our  summary  and 
conclusions  regarding  the  model’s  effectiveness  and  recommend  improvements  and 
further  research. 
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II.  LITERATURE  REVIEW 


A.  EVOLUTIONARY  ACQUISITION 

When  the  DoD  procures  or  develops  a  major  weapon  system  or  platform,  the 
project  is  categorized  as  a  Major  Defense  Acquisition  Program  (MDAP).  The  preferred 
strategy  of  program/project  management  for  these  procurement  endeavors  is  called 
evolutionary  acquisition  (EA).  EA  is  iterative  and  concedes  that  the  nature  of  technology 
is  continually  developing.  By  allowing  major  weapon  systems  to  be  developed  in  steps,  the 
DoD  attempts  to  provide  end  users  with  functional,  relevant  equipment  that  is  supportable 
and  can  be  manufactured  at  a  reliable  rate.  In  EA,  users  employ  the  latest  supportable 
version  of  a  weapon  or  vehicle  while  the  DoD  is  works  to  develop  the  next  version.  For 
MDAPs,  these  updates  are  called  block  upgrades.  Each  time  an  MDAP  is  upgraded,  the 
goal  is  to  add  capability,  relevance,  utility,  and  reliability.  The  EA  strategy  works  on  small 
to  medium  programs  such  as  personal  weapon  systems  and  ground/air  combat  platfonns. 
EA  cannot  be  applied  to  large  and  intricate  systems  such  as  aircraft  carriers  or  submarines. 
For  systems  of  that  nature,  upgrades  represent  an  overhaul  of  the  previous  system  design 
because  they  have  extremely  high  effort-to-production  unit  ratios  and  because  each  new 
increment  or  evolution  embodies  a  large  number  of  new  requirements.  These  platfonns  are 
manufactured  at  immense  cost  and  either  in  small  batches  or  one  at  a  time.  Finally, 
because  these  platfonns  remain  in  near  constant  service  for  three  to  four  decades,  it  is 
cheaper  to  build  entirely  new  ships  rather  than  to  overhaul  the  existing  ones  with  all  of  the 
new  technologies  (“Evolutionary  Acquisition  Strategy,”  2005). 

Because  EA  is  iterative,  accurate  estimation  of  overall  program  cost  is  very 
difficult.  Since  MDAPs  can  span  multiple  decades,  unforeseen  economic  and 
geopolitical  factors  can  cause  increases  in  program  length  and  cost  that  were  originally 
not  factored  into  calculations.  For  this  reason,  Congress  has  had  some  reservations  about 
the  DoD’s  adoption  of  EA  as  its  preferred  strategy  for  weapon  systems  acquisition 
(Lorell,  Lowell,  &  Younassi,  2006). 
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The  major  benefit  of  EA  is  speed  of  delivery.  By  accepting  that  the  perfect 
solution  may  be  unattainable  or  decades  in  the  making,  EA  aims  to  place  the  best  current 
solution  into  the  user’s  hands  as  quickly  as  possible.  Rather  than  waiting  empty-handed 
for  perfection,  users  can  continue  to  work  while  the  DoD  improves  what  is  available  to 
them.  Additional  benefits  of  EA  include  the  potential  to  control  cost  growth  and 
technical  risk.  Acquisition  programs  utilizing  EA  strategy  are  able  to  more  accurately 
estimate  short-term  cost  because  the  program  is  broken  into  smaller  stages  (block 
upgrades).  Finally,  there  is  a  great  deal  of  developmental  flexibility  associated  with  EA. 
Because  the  overall  program  is  segmented  into  block  upgrades,  the  developer  has  more 
time  to  gather  real-world  information  about  the  current  version  in  the  field.  While  user 
feedback  comes  in,  the  developer  is  free  to  make  refinements  to  the  next  version  as  it  is 
being  developed.  This  strategy  has  been  applied  successfully  to  systems  such  as  the  M- 
16  rifle,  Abrams  main  battle  tank,  and  the  F/A-18  Hornet  (“Evolutionary  Acquisition 
Strategy,”  2005). 

Another  example  of  EA  is  the  DoD’s  acquisition  of  the  AH -64  Apache  attack 
helicopter.  First  fielded  in  1983,  the  Apache  has  now  undergone  five  versions  and  three 
block  upgrades.  The  Apache  Block  III  (AB3)  program  uses  the  AH-64D  Longbow  as  a 
starting  point  for  system  upgrades  and  adds  significant  advancements  in  flight  capability 
with  the  General  Electric  701-D  model  engine  and  the  Rotorcraft  Drive  System  for  the 
21st  Century  (RDS-21).  Combined  with  the  High  Performance  Shock  Strut  (HPSS) 
system  and  new  lighter,  faster,  stronger  composite  rotor  blades,  the  RDS-21  allows  the 
latest  Apache  more  combat  capability  while  regaining  the  maneuverability  and  hard 
landing  capabilities  of  the  first  generation  Apache.  Additionally,  the  AB3  will  allow  the 
co-pilot  gunner  (CPG)  to  assume  flight  control  and  view  feed  from  nearby  UAVs.  This 
capability  has  been  named  manned-unmanned  teaming  or  “MUM”  for  short,  and  will 
assist  the  Apache  crew  in  developing  the  tactical  situation  prior  to  arriving  at  an  area  to 
support  ground  operations  (Osbom,  2012,  para.  10). 
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B.  WEAPON  SYSTEMS  ACQUISITION  REFORM  ACT  OF  2009 

After  passing  through  the  House  of  Representatives  and  the  Senate  unanimously, 
the  Weapon  Systems  Acquisition  Refonn  Act  of  2009  was  signed  into  law  by  President 
Barack  Obama.  This  law  makes  multiple,  sweeping  refonns  to  the  DoD  acquisition 
system.  Chief  among  these  changes,  the  law  installs  a  director  of  cost  assessment  and 
program  evaluation  (CAPE)  within  the  DoD.  The  CAPE  director  reports  directly  to  the 
secretary  and  undersecretary  of  defense  and  is  charged  with  issuing  policy  on  cost 
estimation  and  the  confidence  levels  related  to  those  cost  estimates.  The  act  absorbs  the 
Office  of  Program  Analysis  and  Evaluation  into  the  CAPE,  enabling  the  director  to  better 
develop  new  policies  to  bring  policies  on  cost  estimating  back  into  line  with  congressional 
and  executive  guidance  (Weapon  Systems  Acquisition  Refonn  Act  of  2009). 

In  addition  to  establishing  the  CAPE,  the  Weapon  Systems  Acquisition  Refonn 
Act  of  2009  also  amends  the  Nunn-McCurdy  Act  to  allow  the  secretary  of  defense  to 
rescind  a  previously  granted  milestone  approval  in  the  event  that  an  acquisition  program 
in  question  experiences  severe  cost  overruns  (Weapon  Systems  Acquisition  Refonn  Act 
of  2009). 

C.  COST  TRADE-OFFS  IN  ACQUISITION  PROGRAMS 

In  November  2012,  the  USD(AT&L)  Frank  Kendall  released  a  memorandum  that 
stated  his  intention  to  further  refine  the  DoD’s  acquisition  system.  The  memorandum, 
titled  Better  Buying  Power  2.0,  lists  seven  major  objectives  designed  to  improve  the  DoD 
acquisition  workforce’s  decision-making  skills,  enhance  and  sustain  their  professional 
development,  and  maximize  value  in  every  acquisition  program  the  DoD  pursues. 

The  first  objective  in  Kendall’s  (2012)  memorandum  is  titled  “Achieve 
Affordable  Programs.”  In  this  portion  of  the  memorandum,  Kendall  (2012)  illustrates  the 
need  to  prioritize  system  requirements  and  to  make  cost  trade-offs  in  order  to  keep  all 
procurements  within  budget.  By  prioritizing  system  capabilities  and  performing  cost 
trade-offs  for  individual  upgrades  within  block  upgrade  programs,  PMs  may  be  able  to 
meet  the  USD(AT&L)’s  intent. 
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D.  GAO  REPORT  OUTLINING  THE  IMPACT  OF  SEQUESTRATION  ON 

MDAPS 

The  Government  Accountability  Office  (GAO)  Report  12-400SP,  dated  March 
29,  2012,  examined  a  selection  of  the  96  weapon  systems  currently  in  the  DoD 
acquisition  portfolio.  The  report  revealed  that  in  FY  2012,  program  costs  grew  by  5%,  or 
$74.4  billion.  Of  this  cost  overrun,  42%  were  attributed  to  production  inefficiency,  40% 
was  blamed  on  quantity  changes,  and  the  final  18%  was  credited  to  cost  growth  in 
research  and  development  (GAO,  2012). 

The  report  also  highlighted  13  future  programs  and  contrasted  them  with  some  of 
the  MDAPs  that  are  responsible  for  high  cost  growth.  In  favorable  contrast,  the  GAO 
report  illustrated  that  these  programs  are  working  within  the  parameters  set  forth  in  the 
Weapons  Systems  Acquisition  Reform  Act  of  2009  by  emplacing  affordability  targets 
within  their  budget  planning.  This  is  also  in  keeping  with  the  guidance  set  forth  by  the 
USD  (AT&L)’s  Better  Buying  Power  2.0  memorandum.  Additionally,  the  report 
commended  these  projects  for  their  emplacement  of  “should  cost  analysis”  in  their 
decision-making  procedure  (GAO,  2012). 

It  is  the  position  of  this  research  project  that  utility  modeling  can  help  PMs  to 
apply  the  parameters  set  forth  by  Better  Buying  Power  2.0.  If  PMs  within  the  DoD 
acquisition  system  were  able  to  quantitatively  assess  which  individual  upgrades  provided 
the  most  value  to  their  systems,  then  the  acquisition  workforce  would  be  further  enabled 
to  maximize  the  value  derived  from  each  block  upgrade  to  MDAPs.  Unfortunately,  it  is 
very  complicated  and  tune  consuming  to  assemble  all  of  the  pertinent  schedule,  cost,  and 
quality  information  in  order  to  make  a  simultaneous  comparison  of  all  prospective  system 
upgrades  that  are  presented  to  a  PM. 

E.  INTEGER  PROGRAMMING  AND  UTILITY  MODELING 

In  their  textbook  Managerial  Decision  Modeling  with  Spreadsheets,  Balakrishnan 
et  al.  (201 1)  defined  integer  programming  as  a  mathematical  method  that  is  used  to  solve 
complex  problems  involving  multiple  inputs,  constraints,  and  desired  results.  When 
writing  an  integer  program,  the  first  objective  is  to  develop  an  objective.  The  objective  is 
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used  to  develop  the  rest  of  the  model.  Usually,  the  objective  is  either  to  maximize  or 
minimize  a  variable  (cost,  profit,  time,  etc.).  Once  the  objective  has  been  developed,  the 
next  step  is  to  program  the  decision  variables.  These  variables  represent  all  of  the 
questions  that  the  integer  program  intends  to  answer.  These  questions  are  most  often 
found  in  binary  (yes  or  no)  or  quantitative  variables  (how  many).  After  the  decision 
variables  have  been  created,  the  programmer  can  derive  the  objective  function. 
Essentially,  the  objective  function  is  a  mathematical  illustration  of  how  all  decision 
variables  will  holistically  affect  the  objective  of  the  integer  program.  Finally,  the 
program  requires  the  installation  of  constraints.  These  act  as  arithmetic  boundaries  that 
help  to  shape  an  optimal  solution  to  the  overall  problem  that  the  integer  program  seeks  to 
solve.  In  a  production  environment,  constraints  usually  consist  of  things  like  tune, 
materiel,  budget,  and  so  forth  (Balakrishnan  et  al.,  2011). 

When  programmed  accurately  and  used  properly,  integer  programs  can  assist  in 
solving  complex  problems  in  a  short  amount  of  time.  Because  integer  programs  can 
analyze  and  compare  many  different  variables  and  entry  arguments,  as  well  as 
simultaneously  consider  all  the  constraints  that  have  been  programmed,  they  can  save 
decision-makers  a  great  deal  of  tune  and  money.  However,  integer  programs  are  not  a 
catch-all  solution  to  quantitative  problems.  They  are  helpful  in  guiding  a  manager  to  an 
informed,  quantitative  analysis  of  a  given  question,  but  integer  programs  should  not  be 
used  in  a  vacuum.  Instead,  these  models  should  be  used  in  concert  with  other  proven 
decision-making  and  analytical  tools  to  help  guide  a  manager  to  the  best  possible 
production  decision  for  his  or  her  organization  (Balakrishnan  et  al.,  2011). 

F.  PROJECT  MANAGEMENT  IN  THE  DEFENSE  ACQUISITION 

ENVIRONMENT 

One  of  the  main  responsibilities  of  PMs  is  to  manage  their  project  to  fruition  and 
complete  project  deliverables  within  a  set  of  constraints.  These  constraints  shape  the 
environment  in  which  these  deliverables  are  generated.  Depending  on  the  scope  of  the 
project,  constraints  may  be  numerous  and  complex,  or  simple  and  few.  However,  one  set 
of  constraints  is  constant  regardless  of  a  project’s  size  or  scope.  The  triple  constraints  of 

quality  (cost,  schedule  and  quality)  overshadow  all  other  limitations  that  PMs  work 
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within  and  ultimately  dictate  the  fate  of  projects.  Project  cost,  schedule,  and  quality 
represent  an  unrelenting,  co-dependent  framework  of  considerations  and  decisions  that 
the  PM  must  make.  By  their  very  nature,  when  any  two  of  these  constraints  are 
combined,  they  work  to  balance  the  third.  As  an  example,  cost  and  schedule  constraints 
act  to  limit  the  amount  of  quality  that  can  be  achieved  by  a  project.  To  further  illustrate, 
if  a  stakeholder  proposes  additional  quality  for  the  deliverable,  then  either  cost  or 
schedule  (sometimes  both)  must  be  increased. 

In  the  2012  edition  of  their  book  Project  Management  for  Engineering,  Business 
and  Technology,  John  Nicholas  and  Hennan  Steyn  listed  more  than  10  factors  that 
contribute  directly  to  the  quality  of  a  civilian  project  deliverable.  Many  of  these  factors 
readily  translate  into  defense  acquisition  programs/projects.  Areas  such  as  system  safety, 
reliability,  adaptability,  logistic  supportability,  negative  trade-offs,  and  environmental 
impacts  all  represent  significant  concern  to  any  successful  military  acquisition  PM. 
However,  each  potential  upgrade  brings  with  it  a  unique  combination  of  factors  that  must 
be  carefully  considered.  PMs  must  consider  the  quality  implications  of  each  individual 
system  upgrade  separately  from  the  weapon  system  as  a  whole.  Once  PMs  fully 
understand  what  an  individual  upgrade  brings  to  the  table,  they  must  consider  how  well 
its  benefits  and  drawbacks  mesh  with  the  platform  and  program  holistically  (Nicholas  & 
Steyn,  2012). 

With  respect  to  the  AB3  Modernization  Program,  PMs  have  a  great  deal  of 
individual  upgrades  to  consider.  While  looking  at  the  benefits  and  drawbacks  to  each 
upgrade,  PMs  must  also  consider  funding  and  production  schedule  implications. 
Although  a  new  upgrade  may  bring  substantial  capability  to  the  AB3  platfonn,  if  the  cost 
or  schedule  impacts  are  too  extensive,  then  PMs  cannot  sponsor  the  upgrade  for  funding. 
Significant  upgrades  have  already  gained  funding  through  the  POM  process,  such  as  the 
RDS-21  system  or  the  introduction  of  the  General  Electric  Model  701-D  engines 
(Osbom,  2010).  However,  PMs  are  still  considering  several  additional  upgrades  to  add  to 
the  AB3.  As  described  previously,  PMS  must  weigh  the  individual  and  holistic  benefits 
presented  by  these  upgrades  against  their  impact  to  the  PMs’  overall  production  budget 
and  schedule.  Finally,  PMs  must  develop  a  quantity  recommendation  for  each  new 
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upgrade  that  will  be  proposed.  Program  Executive  Office  (PEO)  Aviation  requires  the 
PM  of  the  AB3  program  to  accompany  each  upgrade  recommendation  with  a  proposed 
quantity  of  aircraft  to  upgrade.  This  requirement  underlines  the  need  for  the  PM  to 
perform  in-depth  quantitative  analysis  of  the  downstream  effects  that  each  upgrade  will 
have  on  the  program’s  overall  budget.  The  following  chapter  discusses  the  methods  we 
used  in  this  research  project  to  develop  a  utility -based,  integer  program  model  to  assist 
PMs  in  these  analyses. 
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III.  UTILITY  MODEL  DESIGN  AND  CONSTRUCTION 


A.  MODEL  ARCHITECTURE 

This  research  project  yielded  an  integer  linear  program  that  is  designed  for  use  by 
PMs.  The  integer  linear  program  is  based  in  Microsoft  Excel  and  utilizes  the  Solver  data 
analysis  tool  to  mathematically  yield  the  most  optimal  results  within  preset  constraints. 
The  objective  function  or  goal  of  the  integer  linear  program  is  to  maximize  the  added 
value  of  all  system  upgrades  that  will  be  procured  as  part  of  a  larger  upgrade  to  an 
MDAP.  Ideally,  the  model  solution  will  give  PMs  an  optimal  mix  of  upgrades  to  select 
and  in  what  quantities.  Essentially,  this  linear  model  helps  PMs  to  answer  the  question, 
which  upgrades  should  I  select  and  how  many  should  I  buy?  While  developing  the 
optimal  solution,  the  model  considers  universal  constraints  that  apply  to  any  DoD 
weapon  system  procurement,  such  as  schedule  or  budget.  The  model  can  also  be  easily 
customized  to  consider  additional  program-specific  constraints  such  as  small  business 
inclusion,  minimum  or  maximum  system  selection  quantities,  and  so  forth,  should  the 
user  need  them  included.  This  model  is  not  intended  to  provide  PMs  with  a  final  answer 
to  any  acquisition  question.  It  is  intended  to  provide  PMs  with  a  quantitatively  derived 
entry  argument.  The  results  of  the  model  can  also  be  used  to  “check”  results  of  a 
previous  analysis  to  determine  whether  a  block  upgrade  program  is  on  track  to  provide 
the  best  added  value  to  the  MDAP.  The  tool  consists  of  three  integrated  systems.  These 
systems  work  together  to  interface  with  PMs,  to  deliver  PM  inputs  to  the  integer  linear 
program,  and  to  perform  the  integer  linear  program  calculations. 

B.  EXCEL  WORKSHEET 

In  designing  and  creating  the  utility  model,  our  focus  was  to  develop  relevant 
inputs  and  an  efficient  method  to  organize  them.  In  order  to  reduce  the  amount  of  models 
required,  we  designed  an  Excel  worksheet  to  serve  as  an  intennediary  data  entry 
platform.  We  programmed  the  worksheet  to  calculate  the  overall  objective  function  and 
to  organize  other  relevant  inputs  in  a  manner  that  can  be  easily  used  when  inputting  data 
into  the  utility  model.  The  most  difficult  part  of  designing  the  worksheet  and  the  utility 
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model  was  to  determine  what  the  objective  function  would  be.  According  to 
Balakrishnan  et  al,  (2011)  the  objective  function  is  “a  mathematical  statement  of  the  goal 
of  an  organization,  stated  as  the  intent  to  maximize  or  minimize  some  important  quantity” 
(p.  53).  The  goal  of  the  utility  model  is  to  determine  the  optimal  allocation  of  upgrade 
alternatives  to  achieve  best  overall  value.  As  a  result,  the  objective  function  is  to 
maximize  the  overall  value  of  the  upgrade  alternatives.  According  to  the  Federal 
Acquisition  Regulation,  “Best  means  the  expected  outcome  of  an  acquisition  that,  in  the 
Government’s  estimation,  provides  the  greatest  overall  benefit  in  response  to  the 
requirement.”  (Federal  Acquisition  Regulation,  2005,  sec.  2.101).  To  derive  the  overall 
benefit,  cost,  schedule,  and  quality  factors  must  be  considered  to  determine  the  ultimate 
benefit.  The  utility  model  compares  each  of  these  to  attributes  from  each  potential 
upgrade  to  find  the  optimal  allocation  of  funding  resources  and  achieve  the  best  overall 
value  for  the  taxpayer. 

The  Quality  category  is  the  most  subjective  and  critical  aspect  of  the  utility 
model.  According  to  the  Defense  Acquisition  Guidebook  (DoD,  2013),  “Quality  is  the 
degree  to  which  a  set  of  inherent  characteristics  fulfills  requirements.”  Quality  is  such  an 
essential  aspect  in  determining  the  best  overall  value  that  we  centered  the  objective 
function  score  of  the  utility  model  around  it.  Quality  can  be  subjective,  and  in  order  for 
quality  to  be  incorporated  into  the  utility  model,  it  must  be  quantified.  The  worksheet  is 
the  tool  to  quantify  the  subjective  worth  of  the  new  or  improved  capability  presented  by 
the  upgrade.  Utilizing  the  worksheet,  PMs  can  derive  the  objective  function  score  or 
what  is  called  the  “quality  index  score”  (QIS).  The  QIS  is  the  score  given  to  each 
upgrade  that  will  be  used  to  compare  it  against  all  others.  Determining  the  QIS  is  the  art 
and  science  of  program  management,  and  making  this  determination  relies  heavily  on  PM 
input.  In  the  case  of  the  AB3  Modernization  Program,  we  examined  14  separate  quality 
categories  to  determine  the  QIS.  Considering  more  quality  factors  will  result  in  a  more 
complete  QIS.  The  remainder  of  the  worksheet  is  used  to  organize  competitive  influence 
factors,  such  as  weighting  and  smoothing  coefficients,  and  cost  in  a  manner  that  is  easy  to 
input  into  the  utility  model.  Cost  is  considered  a  constraint  and  is  factored  against  the 
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QIS  in  the  utility  model.  Without  the  generation  of  the  QIS  within  the  worksheet,  the 
utility  model  cannot  be  used. 

1.  COMPETITIVE  INFLUENCE  FACTORS 

The  first  category  of  the  worksheet  is  the  Competitive  Influence  Factors  (CIF) 
section.  As  Figure  1  shows,  the  CIF  section  is  where  the  upgrades  are  listed  across  the 
first  row  of  the  spreadsheet.  Upgrades  A  through  E  in  the  example  in  Figure  2  represent 
the  various  upgrades  to  be  considered.  This  section  is  critical  to  determining  the  overall 
quality  of  the  potential  upgrade. 
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%  Improved  Capability 

Weights  sum  to 


Smoothing 


Figure  1.  Competitive  Influence  Factors 


Weighting:  The  first  CIF  factor  in  determining  the  best  overall  value  is  the 

upgrade’s  weight.  Some  upgrades  may  bring  such  a  critical  capability  that  they  are  given 

more  importance  to  other  potential  upgrades  relative  to  the  decision-maker.  Here,  PMs 

are  allowed  to  give  that  critical  upgrade  a  higher  weight  than  others  to  ensure  it  is 

accounted  for  in  the  end  product.  If  all  potential  upgrades  are  equally  valued,  PMs  can 

distribute  the  weights  evenly,  or  simply  leave  the  weight  values  blank.  There  are  various 

methods  to  weight  an  upgrade.  Weights  can  be  listed  in  the  following  fonnats:  in 

decimal  fonn  that  either  does  or  does  not  sum  to  1  or  in  rank  order  (e.g.,  A  =  first  place, 
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B  =  second  place... E  =  fifth  place).  For  example,  Upgrade  A  may  get  a  weight  of  5, 
while  B  and  C  get  2.5,  and  D  and  E  get  1.  It  is  paramount  that  weights  are  assigned  in  a 
unifonn  fonnat  in  order  to  correctly  weight  the  upgrade.  Also,  PMs  must  carefully 
consider  the  weight  scores  they  award  because  weight  plays  a  significant  role  in 
determining  the  final  outcome  of  the  utility  model.  PMs  can  also  assign  priorities  to 
different  upgrades  according  to  their  current  assessment  of  the  upgrade  project.  For 
instance,  PMs  could  assign  a  score  of  3,  2,  or  1,  respectively  to  each  potential  upgrade 
according  to  its  importance  with  respect  to  the  success  of  the  MDAP  block  upgrade. 

Smoothing  Constant:  The  smoothing  constant  is  a  value  greater  than  zero  and 
less  than  one  that  is  used  to  smooth  out  abrupt  exponential  fluctuations  so  that  the  model 
provides  stable  estimates.  The  higher  the  smoothing  constant  is,  the  smoother  the  total 
final  score  will  be.  If  PMs  are  more  interested  in  an  aggressive  upgrade  portfolio,  then 
they  can  utilize  a  lower  smoothing  constant.  For  a  more  balanced  upgrade  portfolio,  PMs 
would  choose  a  higher  smoothing  constant. 

Percent  Improved  Capability:  This  is  the  overall  improvement  the  proposed 
upgrade  contributes  to  the  system’s  current  state.  A  33%  improvement  is  listed  in  whole 
numbers  as  33  in  order  to  ensure  its  value  is  accounted  for  in  the  overall  scoring  of  the 
upgrades. 

The  next  portion  of  the  worksheet  is  concerned  with  the  organization  of  the 
influence,  quality,  cost,  schedule,  and  quantity  categories.  Figure  2  displays  how  we 
arranged  these  categories  in  conjunction  with  the  listed  potential  upgrades.  The  top 
categories  are  influencing  and  quality  categories,  while  cost,  schedule,  and  quantity  are 
constraint  categories. 
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Figure  2.  Worksheet  Layout 


2.  QUALITY  CATEGORIES 

Each  MDAP  block  upgrade  is  unique  and  requires  a  variety  of  skills  and 
resources  to  successfully  complete.  Therefore,  each  block  upgrade  program  must  have 
unique  factors  to  be  considered  in  order  to  determine  the  quality  of  an  upgrade.  Within 
the  Quality  category,  some  basic  quality  factors  should  always  be  considered,  along  with 
the  unique  aspects  of  each  program.  These  factors  include  perfonnance,  safety, 
reliability,  ergonomics,  maintainability,  logistical  support,  environmental  considerations, 
increased  capability  over  the  current  state,  technology  readiness  level  (TRL),  and  small 
business  employment.  It  is  important  to  understand  that  the  model  we  created  in  this 
research  is  not  intended  to  compare  upgrades  that  perfonn  similar  tasks;  rather,  our  utility 
model  is  designed  to  compare  disparate  upgrades  that  provide  unique  capabilities.  The 
purpose  of  the  utility  model  is  to  identify  trade-offs  between  technologies  that  are  being 
considered  in  a  block  upgrade  program.  Within  the  quality  category,  PMs  and  their  staff 
must  develop  as  complete  a  list  of  quality  factors  as  possible  in  order  to  achieve  the  most 
accurate  and  realistic  QIS.  The  QIS  (objective  function)  is  based  on  the  quality  category. 
As  stated  earlier,  cost  and  schedule  factors  are  considered  constraints;  therefore,  ensuring 
a  complete  understanding  of  quality  is  critical  to  obtaining  the  optimal  trade-off.  It  must 
be  noted  that  in  some  circumstances  schedule  can  be  used  as  a  factor  of  quality  rather 
than  as  a  constraint.  If  PMs  are  making  simultaneous  assessments  of  potential  upgrades, 
all  of  which  promise  an  implementation  timeline  that  fits  within  the  PMs’  schedules,  then 
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the  PMs  must  instead  assess  which  upgrades  make  the  most  economic  use  of  their  tune. 
The  use  of  schedule  as  a  factor  of  quality  allows  PMs  to  frontload  the  upgrades  that 
provide  quality  to  the  project  in  a  timelier  manner.  We  discuss  schedule  as  a  quality 
factor  in  further  depth  in  the  next  section. 

3.  QUALITY  SCORING  METHODS 

Each  category  listed  in  the  quality  portion  of  the  worksheet  is  unique  and  may 
need  to  be  rated  in  a  unique  manner.  Table  1  lists  several  methods  for  rating  a  quality 
category  in  order  to  ensure  an  appropriate  comparison  of  upgrades.  A  point  system  is 
used  to  compare  each  upgrade.  When  using  the  various  rating  types,  a  point  total  is 
determined  based  on  PM  inputs  and  assessments.  These  are  summed  at  the  completion  of 
the  evaluation  process,  and  the  QIS  is  determined  for  each  potential  upgrade. 


Table  1.  Rating  Scales 


1  to  10  Scale 

In  this  data  type,  a  scale  from  1  to  10  is  used  to  assess 
quality.  Since  maximizing  overall  value  is  the  goal,  10 
is  the  highest  rating  for  a  quality  category  and  1  is  the 
lowest. 

Yes/No  (Binary) 

In  this  example,  1  =  yes  and  0  =  no.  This  data  type  is 
used  in  cases  such  as  a  small  business  consideration  or 
when  determining  whether  the  upgrade  meets  a 
threshold.  A  yes  answer  receives  1  point  and  a  no 
answer  receives  0  points. 

Percentage 

This  data  type  is  used  for  increments,  such  as  percentage 
of  increased  capability.  For  example,  if  a  new  radar 
system  increases  the  effective  range  from  3  km  to  4  km, 
then  a  33%  increase  in  capability  is  added.  This  is 
recorded  as  33  points. 

Subtracting  Factors 

This  data  type  is  used  to  subtract  points  in  creating  a 
negative  impact  to  the  current  system  in  exchange  for  its 
new  capability.  For  example,  by  adding  extra  annor  to  a 
vehicle,  the  fuel  mileage  and  maneuverability  of  the 
vehicle  are  diminished.  In  this  example,  1  =  highest 
negative  impact  (least  positive)  and  10  =  lowest  negative 
impact  (most  positive). 
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4.  QUALITY  INDEX  SCORE  (QIS) 

To  determine  the  QIS,  the  worksheet  calculates  the  sum  of  all  quantities  in  the 
Quality  category.  The  sum  of  the  quality  categories  is  multiplied  by  the  weight  assigned 
in  the  Competitive  Influence  Factors  section.  This  is  done  for  each  potential  upgrade. 
Figure  3  displays  the  Excel  fonnula  for  QIS  calculation. 
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Upgrade 

Item  A 
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Influence  Factors 
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Smoothing  Coefficient 
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7 

Quality  Category 

8 
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Quality  Category 
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11 

Quality  Category 

Quality  Index  Score 

=[(SU  M(C4:C10+C4)  *(1+C2)]*C3 

QIS 


Figure  3.  QIS  Calculation 


5.  SCHEDULE  CONSTRAINTS 

The  schedule  portion  of  the  worksheet  is  used  to  annotate  the  impact  the  upgrade 
will  have  to  the  overall  project  either  in  terms  of  length  of  time  to  install  the  upgrade  to 
the  necessary  units  or  in  impacts  to  the  PMs’  personnel  managing  the  project.  If 
measuring  the  length  of  time  to  install  the  upgrade,  worksheet  users  should  input  the 
standard  time-measuring  units  that  are  used  by  PMs.  For  example,  weeks,  months, 
quarters,  or  years  could  be  listed.  If  listing  the  impacts  to  the  PMs’  staff  in  terms  of 
managing  the  project,  worksheet  users  should  use  the  standard  time-measuring  metric 
that  is  used  within  that  PMO  in  other  decision-making  vehicles.  For  example,  man  years, 
weeks,  months,  or  quarters  of  years  could  be  used.  Ensuring  that  the  same  time  metric  is 
unifonnly  applied  to  all  upgrades  is  critical  to  producing  a  relevant  comparison. 
Worksheet  users  may  have  to  convert  units  of  time  to  ensure  they  are  being  measured 
equally.  For  example,  if  one  project  takes  two  years  to  fully  implement  and  another  takes 
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six  months,  then  worksheet  users  may  have  to  use  24  months  or  0.25  years  to  ensure  time 
measurement  is  consistent. 

In  many  cases,  PMs  are  not  responsible  for  the  actual  production  of  the  upgrade 
item  that  is  being  added  to  the  MDAP  (typically  this  is  the  contractor’s  responsibility).  In 
such  cases,  the  time  it  takes  to  produce  the  item  then  becomes  a  quality  factor  rather  than 
a  constraint.  If  evaluating  several  competing  upgrades,  shorter  development  time  is 
better;  therefore,  this  becomes  a  quality  factor  that  should  be  accounted  for.  To  account 
for  schedule  impacts  (also  known  as  cumulative  retrofit  time)  within  the  QIS,  a  change  to 
the  overall  QIS  formula  is  required.  To  simplify  accounting  for  such  items,  worksheet 
users  should  simply  place  into  the  model  the  amount  of  time  (unifonnly  measured  for  all 
upgrades)  it  takes  to  retrofit  the  end  item.  For  example,  24  is  entered  for  24  months 
within  the  quality  factors.  Since  a  longer  retrofit  tune  is  less  desirable,  this  must  count 
against  the  overall  QIS  for  the  upgrade  in  question.  The  new  fonnula  accounting  for  the 
retrofit  tune  is  listed  in  Equation  1. 

QIS  =  [(X  Quailty  Factors  +  %  Improved  Capability j  *  (1  +  Upgrade  Weight)] 

*  Smoothing  Coefficient 

QIS  Formula  1 

6.  COST  CONSTRAINTS 

Cost  is  measured  in  dollars  and  should  be  comprised  of  the  projected  program 
acquisition  unit  cost  (PAUC).  Figure  4  shows  how  this  should  be  listed  in  the  worksheet. 


Total  Budget 

Upgrade  A  PAUC 

Upgrade  B  PAUC 

Cost 

Upgrade  C  PAUC 

Upgrade  0  PAUC 

Upgrade  E  PAUC 

Figure  4.  Program  Acquisition  Unit  Cost  Constraint 
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7. 


QUANTITY  CONSTRAINTS 


Within  the  Quantity  category,  PMs  will  list  the  minimum  and  maximum  units  that 
they  are  able  to  purchase  for  each  program.  The  quantity  constraint  will  ensure  that  the 
PMs’  ceiling  and  floor  quantities  are  factored  and  measured  in  terms  of  units  purchased. 
Minimums  and  maximums  are  listed  together  to  simplify  the  data  input  into  the  utility 
model. 


Quantity 

Minimum  Quantity  A 

Minimum  Quantity  B 

Minimum  Quantity  C 

Minimum  Quantity  D 

Minimum  Quantity  E 

Maximum  Quantity  A 

t  > 

Min  and  Max  Units  for 

each  Upgrade 

- 

Maximum  Quantity  B 

Maximum  Quantity  C 

Maximum  Quantity  D 

Maximum  Quantity  E 

Figure  5.  Unit  Quantity  Constraint 


8.  PROGRAM  UNIQUE  CONSTRAINTS 

Other  than  cost,  schedule,  and  quantity  constraints,  there  may  be  times  when 
additional  unique  constraints  are  required.  There  are  three  common  alternative 
constraints  that  are  most  often  utilized.  Interdependency  constraints,  such  as  the  ones 
listed  in  Table  2,  can  be  programmed  into  the  integer  linear  program  portion  of  the  utility 
model  and  can  be  used  by  PMs. 
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Table  2.  Unique  Constraint  Types 


Constraint  Type 

Example 

Selecting  k  of  n  choices 

For  use  when  a  certain  amount  of  choices  from  a  total 
amount  must  be  selected.  For  example,  out  of  five  choices, 
at  least  three  must  be  selected:  X±  +  X2  +  X3  +  X4  + 
X s  >  3. 

Mutually  exclusive 
choices  or  avoiding 
incompatibilities 

This  type  of  constraint  can  be  used  in  two  scenarios. 

Scenario  1 :  No  more  than  one  upgrade  can  be  installed  from 
a  group.  An  example  of  this  would  be  selecting  from 
upgrades  that  perform  similar  functions,  such  as  choosing  a 
type  of  tire  from  several  tire  options.  Once  a  tire  is  selected, 
there  is  no  longer  a  tire  requirement.  X1  +  X2  <  1. 

Scenario  2:  Selecting  a  specific  upgrade  ensures  that  some 
other  upgrade  cannot  be  selected.  In  this  case,  the  upgrades 
are  mutually  exclusive.  For  example,  if  a  wheeled  chassis  is 
selected,  a  tracked  chassis  cannot  be  used.  X1  +  X2  —  1. 

If-then  (linked)  choices 

If-then  choices  means  that  if  Upgrade  X±  is  selected,  then 
Upgrade  X2  must  also  be  selected.  For  example,  additional 
radios  are  installed,  then  a  larger  alternator  must  also  be 
installed.  X±  <  X2  shows  a  one-way  linkage  in  that  if  A  is 
installed,  then  D  must  also  be  installed,  but  not  vice  versa. 

C.  INTRA-MODEL  DATA  FLOW 

As  PMs  answer  the  questions  posed  within  the  Excel  worksheet,  the  data  entered 
are  referenced  to  the  integer  linear  program.  As  an  example,  when  PMs  use  the  data  field 
in  the  worksheet  to  assign  a  name  to  an  upgrade  program  under  consideration  (e.g.,  High 
Perfonnance  Shock  Strut),  the  same  information  replaces  a  generic  placeholder  (e.g., 
Upgrade  A)  with  the  name  of  the  upgrade.  The  worksheet  now  “understands”  that  the 
first  upgrade  under  consideration  is  called  High  Perfonnance  Shock  Strut.  All 
infonnation  provided  by  PMs  regarding  the  High  Performance  Shock  Stmt  is  input  and 
relayed  from  the  worksheet  to  the  correct  place  within  the  integer  linear  program. 

As  data  is  inputted  into  the  worksheet,  they  are  simultaneously  “copied”  and 
relayed  to  the  integer  linear  program  for  computation.  The  title  of  the  first  decision 
variable  began  its  life  as  Upgrade  A  and  has  now  been  transformed  into  High 
Performance  Shock  Strut  on  both  the  worksheet  and  the  integer  linear  program. 
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Additionally,  the  automated  data  relay  sends  QIS  values  and  constraint  information  from 
the  worksheet  into  the  integer  linear  program  for  computation. 

To  accommodate  numerous  or  unique  constraints,  this  research  project 
constructed  room  for  growth  within  the  utility  model  to  accommodate  changes  in 
constraints  and  variables  as  an  MDAP  program  changes.  Specifically,  the  worksheet  has 
space  for  up  to  20  potential  upgrades.  PMs  can  also  provide  program-unique  constraints, 
such  as  minimum  or  maximum  procurement  quantities  or  upgrade  dependencies.  There 
are  empty  relational  cells  within  the  worksheet  and  the  integer  linear  program  that  are 
ready  to  accommodate  this  data.  If  this  additional  or  unique  information  is  entered  in  the 
worksheet,  it  will  be  relayed  to  the  integer  linear  program  and  will  factor  into 
computations.  If  no  data  is  entered,  the  relational  cells  will  remain  empty  and  will  have 
no  impact  on  computation. 

D.  INTEGER  LINEAR  PROGRAM 

The  Excel-based  integer  linear  program  is  simply  the  calculation  engine  for  the 
quantitative  assessment  of  the  utility  model.  This  research  project  developed  it  as  a  shell 
that  can  be  customized  to  fit  the  decision  variables  and  constraints  of  many  different 
types  of  MDAP  block  upgrade  programs.  The  integer  linear  program  consists  of  an 
objective  function,  decision  variables,  and  constraints.  The  objective  function  reflects  the 
QIS  scores  that  are  developed  by  the  PM  within  the  worksheet  tab  and  is  the  “answer” 
that  the  integer  linear  program  must  seek  to  maximize.  It  provides  a  quantitative  solution 
that  tells  PMs  how  many  of  each  upgrade  to  purchase  in  order  to  provide  the  best  overall 
value  to  stakeholders. 

Before  the  objective  function  can  provide  an  answer,  the  integer  linear  program 
must  receive  decision  variables  from  the  Excel  worksheet.  These  decision  variables  will 
be  provided  to  the  integer  linear  program,  complete  with  QIS  values  that  tell  the  model 
how  much  value  they  actually  provide.  Along  with  the  QIS,  each  decision  variable  is 
assigned  cost  and  schedule  constraints.  Additionally,  PMs  can  add  any  project-unique 
constraints  such  as  quantity  minimum  and  maximums  and  dependencies.  Through  cell 
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referencing,  the  worksheet  will  “tell”  the  integer  linear  program  how  long  each  upgrade 
will  take  to  complete  and  how  much  it  will  cost. 

Finally,  the  model  will  take  into  consideration  any  constraints  that  provide 
boundaries  for  a  feasible  solution.  Along  with  cost  and  schedule  restrictions,  any 
additional  constraints  or  dependencies  that  have  been  provided  by  the  PM  will  make  their 
way  into  the  model  and  will  be  factored  into  the  decision  of  the  objective  function.  Once 
all  information  has  been  entered  and  PMs  request  computation,  Excel  will  utilize  the 
Solver  extension  to  find  the  optimal  solution.  The  solution  is  accompanied  by  the  QIS 
information  that  has  been  entered  by  PMs.  These  data  ann  PMs  with  finite  quantities  of 
upgrades  to  purchase  as  well  as  quantitate  value-added  information.  This  data  can  either 
assist  in  the  decision-making  process  or  serve  as  a  check  against  the  PMs’  current 
selection  and  decision-making  process.  Figures  6  and  7  display  the  Excel  Solver 
declarations  and  objective  function  solutions,  respectively. 
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Figure  6.  Excel  Solver  Declarations 
27 


Upgrade 

Upgrade  A 

Upgrade  B 

Upgrade  C 

Upgrade  D 

Upgrade  E 

Decision  Variables 

26 

15 
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15 

15 

Quantities  to  Purchase 

Objective  Function  (QIS) 

39.56 

37.72 

45.12 

29 

45 

10465}  Final  Value  Added 

Figure  7.  Objective  Function  Solution 
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IV.  APPLYING  THE  UTILITY  MODEL 


A.  EVALUATED  TECHNOLOGIES 

The  PM  for  the  AB3  Modernization  Program  provided  this  research  project  with 
16  technologies  to  be  evaluated  using  the  utility  model.  These  technologies  allowed  us  to 
test  and  experiment  with  the  utility  model.  Each  of  these  technologies  is  unique  in 
function  and  is  mutually  exclusive.  Those  technologies  highlighted  in  Figure  8  represent 
a  software-related  upgrade.  None  of  these  technologies  were  included  in  the  POM  at  the 
time  we  completed  our  assessment  because  they  were  being  evaluated  by  the  AB3 
Modernization  PM  and  his  staff  for  integration  in  the  AB3  Modernization  Program. 


Decaying  Rotor  Indication 

Opposite  Seat  Fixed  Gun  Message 

CMWS  Indication 

Secure  Communications 

FM  Muting 

Discrete,  Selectable  ASE  Volumes 

Certified  PERF  Page 

Hydraulic  Pressure  Digital  Readouts 

AH-64E  MTADS  Jitter 

Enhanced  Transmission/Dual  Accessory 

Dual  HADS  Failure 

UTA  Weight/Capability  (C,  F,  S,  &  UHF) 

Remote  HF  Safety  Fan  (Display) 

Seat  Design 

TADS  Failure  Weapon  Inhibit 

VHF  Secure  Communications 

Figure  8.  Evaluated  Technologies.  Highlighted  Upgrades  Are  Software -related. 


B.  WORKSHEET  CUSTOMIZATIONS 

To  be  a  useful  tool,  we  had  to  make  our  model  customizable  to  each  program 

using  it.  In  the  case  of  the  AB3  Modernization  Program,  the  majority  of  the  PM’s 

customizations  to  constraints  and  quality  factors  were  done  on  the  worksheet.  As  stated 

in  Chapter  III,  the  more  quality  factors  that  can  be  included  into  the  model,  the  more 

accurate  the  model’s  results.  As  the  model  was  applied  to  the  AB3  Modernization 
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Program,  the  PM  added  two  new  quality  factors  and  adjusted  two  others  from  the  original 
list.  This  increased  the  amount  of  quality  factors  to  14  for  each  of  the  16  technologies  to 
be  evaluated.  Figure  9  lists  the  final  quality  factors  to  be  evaluated  for  the  AB3 
Modernization  Program.  Those  factors  highlighted  in  blue  represent  changed  quality 
factors,  and  those  highlighted  in  yellow  are  new  factors. 


TRL Level 

Contracting  Ease 

Training  Time  Implication 

Negative  Impact 

Safety 

R  e  1  i  a  b  i  1  ity 

Source  Value 

Quality 

Ergonomics 

Adaptability 

Mean  Time  to  Repair  (MTTR) 

M  a  i  nta  i  n  a  b  i  1  ity 

Logistic  Support 

Environmental  Impact 

Cumulative  Retrofit  Time 

Quality  Index  Score 

Figure  9.  AB3  Quality  Factors.  Factors  Highlighted  in  Blue  Are  New  Factors.  Those  in 

Yellow  are  Factors  that  Have  Been  Changed. 

In  the  utility  model’s  test  application,  using  the  AB3  Modernization  Program,  the 
PM  changed  the  Small  Business  quality  factor  to  Contracting  Ease.  This  was  changed 
because  most  small  businesses  cannot  produce  the  technology  on  the  scale  or  complexity 
required  for  the  AB3  Modernization  Program.  The  term  contracting  ease  refers  to  the 
swiftness  that  PMs  can  procure  new  items.  For  example,  it  is  more  beneficial  to  the 
government  if  an  item  can  be  procured  from  a  vendor  who  is  already  familiar  with  the 
project  because  this  procurement  can  be  done  quickly. 
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The  source  value  category  refers  to  the  origin  of  a  requirement  or  idea.  Potential 
upgrades  that  have  spawned  from  the  voice  of  the  customer  are  given  the  highest  quality 
factor  score  because  they  are  considered  an  operational  pull.  Other  sources  of  requirements 
are  working  groups  with  contractors  or  government  integrators  and  are  considered  to  be  a 
technological  push.  Upgrades  that  seek  to  fill  an  operational  pull  are  given  more  value 
because  they  seek  to  meet  an  immediate  need  of  the  customer  in  the  field. 

The  mean  time  to  repair  (MTTR)  category  is  the  average  time  required  to  perfonn 
maintenance  over  a  specific  operating  period.  This  quality  factor  is  heavily  valued  by 
PMs  because  this  time  is  derived  during  the  development  stage  of  the  upgrade;  therefore, 
it  has  more  relevance  to  PMs  when  comparing  various  upgrades  (Jones,  2006). 

The  cumulative  retrofit  time  (CRT)  factor  is  the  time  it  takes  to  integrate  the  new 
technology  onto  the  platfonn.  The  addition  of  this  quality  factor  to  the  worksheet 
resulted  in  the  removal  of  the  schedule  constraint  because  the  PM  for  the  AB3 
Modernization  Program  is  not  responsible  for  actually  building  the  new  technology;  the 
contractor  is.  As  a  result,  the  amount  of  time  to  produce,  install,  and  fully  integrate  the 
item  must  be  accounted  for  as  a  quality  factor.  The  time  in  this  case  is  listed  in  months 
(48  =  48  months  to  produce  and  integrate).  What  must  be  kept  in  mind  is  that  the  higher 
the  score,  the  more  negative  the  impact  assessed  to  the  overall  QIS.  This  is  opposite  to 
the  rest  of  the  scoring  on  the  quality  portion  of  the  worksheet.  Up  to  this  point,  a  higher 
numerical  score  has  always  been  better  to  maximize  the  objective  function.  When 
considering  time  as  a  quality  factor,  the  unique  QIS  fonnula  displayed  in  Equation  2  must 
be  used  to  penalize  QIS  scores  for  upgrades  with  longer  CRTs  and  reward  those  with 
shorter  CRTs. 


QIS  =  Quality  Factors  +  %  Improved  Capability ^  *  (1  +  Upgrade  Weight ) 

—  ( Upgrade  Weight  *  Cumulative  Retrofit  Time)] 

*  Smoothing  Coefficient 


Figure  10.  Unique  QIS  Fonnula 
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C.  UNIQUE  PROJECT  CONSTRAINTS 

In  the  case  of  the  AB3  Modernization  Program,  software-related  upgrades  were  a 
unique  constraint  that  had  to  be  considered.  In  the  AB3  program,  software-related 
upgrade  purchase  decisions  are  binary.  If  the  model  calculated  that  a  single  platfonn 
should  receive  a  software -related  upgrade,  then  each  of  the  790  platforms  had  to  be 
upgraded.  This  decision  is  binary  to  mitigate  the  risk  that  could  occur  if  the  platforms  are 
not  interoperable.  In  this  case,  a  binary  constraint  was  put  in  place  to  ensure  that  if  a 
software  program  was  chosen,  then  each  platfonn  would  receive  the  upgrade.  To 
accommodate  these  unique  decisions,  we  programmed  customizations  into  the  Excel 
Solver  constraints  and  also  into  the  decision  variables  themselves.  These  customizations 
applied  only  to  the  software-related  upgrades  and  allowed  the  utility  model  to  either 
calculate  a  fleet-wide,  790-piece  purchase  decision  or  decline  the  purchase  completely. 

D.  WEIGHTING  METHODOLOGIES 

PMs  can  use  one  of  several  weighting  methods  to  subjectively  apply  their 
preference  to  potential  upgrades.  Weight  scores  are  a  way  for  PMs  to  add  a  measured, 
subjective  influence  into  an  otherwise  dispassionate  equation.  There  are  many  reasons 
why  PMs  may  wish  to  give  certain  upgrades  a  minor  boost  in  a  comparison  with  other 
upgrades.  The  most  valid  reasons  for  such  an  addition  can  be  traced  to  either  operational 
needs  or  personal  experience.  Most  upgrades  originate  in  operational  needs  statements 
(ONSs)  from  forward  deployed  areas  where  end  users  are  finding  shortfalls  with  their 
equipment.  ONSs  are  an  example  of  an  operational  pull  that  originates  from  the  voice  of 
the  customer  (VOC),  or  users  in  the  field.  Weighting  potential  upgrades  allows  PMs  to 
make  a  stronger  case  for  upgrades  that  fill  a  specified  operational  need  within  the 
confines  of  the  model.  Further,  many  PMs  are  assigned  oversight  of  platfonns  that  they 
have  experience  with.  When  we  tested  our  model  by  applying  it  to  the  AB3 
Modernization  Program,  we  found  that  the  PM  was  an  AH-64  Apache  pilot.  This  is 
considered  an  Anny  best  business  practice.  The  Anny  Acquisition  Corps  often  appoints 
personnel  with  direct  platfonn  experience  to  serve  as  PMs.  Personnel  who  have  operated 
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and  managed  a  system  stand  a  much  greater  chance  of  making  valid  and  informed 
decisions  with  respect  to  that  platform. 

For  the  experimental  application  of  this  research  project,  the  weighting 
methodologies  utilized  were  ranking,  summation,  and  prioritization.  Each  of  these 
weighting  methods  has  inherent  advantages  and  drawbacks.  The  ranking  method  works 
well  for  assessments  of  between  five  and  20  potential  upgrades,  but  causes  large 
arithmetic  “gaps”  between  those  potential  upgrades  that  receive  the  highest  and  lowest 
scores.  The  summation  method  works  very  well  for  assessments  of  five  or  fewer 
upgrades.  The  prioritization  method  works  well  for  any  sized  group  of  potential  upgrades 
because  the  resulting  QIS  scores  most  directly  reflect  the  PMs’  subjective  preference. 
However,  during  the  application  of  our  model,  the  AB3  Modernization  PM  did  not  utilize 
a  weighting  method. 

E.  DATA  INPUT  AND  QIS  GENERATION 

With  the  appropriate  customized  quality  factors  in  place,  the  AB3  Modernization 
PM  utilized  the  worksheet  to  make  an  un-influenced  or  optimized  assessment  of  the  16 
potential  upgrades  mentioned  in  Section  A  of  this  chapter.  The  PM  decided  to  take  an 
objective  pass  to  see  what  decisions  the  model  would  make  without  any  subjectivity  in 
place.  In  order  for  the  CRT  quality  factor  to  be  taken  into  account,  a  default  weight  must 
be  entered.  In  the  absence  of  a  default  weight  value,  the  CRT  quality  factor  is  reduced  to 
zero,  negating  its  influence  on  the  QIS  score.  For  an  un-biased  evaluation  of  each 
upgrade,  the  default  weight  value  must  be  unifonn.  This  research  project  recommends  a 
default  weight  value  of  1.  Figure  11  displays  the  QIS  scores  that  were  computed  by  the 
utility  model. 
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Figure  1 1.  AB3  Modernization  Program  Completed  Worksheet:  Control  Data 
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The  QIS  scores  generated  for  the  16  potential  upgrades  ranged  from  55.2  quality 
points  to  117.6.  The  average  QIS  score  was  77.2.  Eleven  potential  upgrades  scored 
below  80,  and  the  remaining  five  scored  between  80  and  117.6  points.  Regression 
analysis  showed  that  the  two  most  influential  quality  factors  in  the  absence  of  PM 
weighting  were  Percent  Improved  Capability  and  TRL  Level. 

F.  UTILITY  MODEL  RESULTS 

In  the  application  of  our  utility  model,  the  AB3  Modernization  PM  was  not 
searching  for  a  satisficing  answer  that  could  be  provided  through  weighted  utility 
modeling  with  the  weighting  methods  listed  previously.  Instead,  he  wanted  an  optimized 
solution  that  represented  the  best  overall  value  for  the  portfolio,  absent  of  exterior 
influence.  To  enhance  the  depth  of  this  research  project,  we  treated  these  optimized 
results  as  a  control  data  set.  We  conducted  three  experiments  in  order  to  ascertain  the 
impacts  of  weighted  utility  modeling  and  the  different  weighting  methodologies  within 
this  utility  model.  The  weighting  methodologies  utilized  in  these  experiments  were 
ranking,  summation,  and  prioritization.  Each  of  these  weighting  methods  has  inherent 
advantages  and  drawbacks.  The  control  assessment  results  from  the  AB3  Modernization 
PM  are  displayed  in  Figure  12. 
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Figure  12.  AB3  Modernization  Program:  Control  Results 
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The  objective  (control)  results  produced  by  the  AB3  Modernization  PM’s 
optimized  assessment  provided  a  somewhat  top-heavy  and  software-biased  portfolio  of 
upgrades.  Each  software-related  upgrade  was  chosen.  Additionally,  hardware  upgrades 
that  were  chosen  included  the  following: 

•  Modernized  Target  Acquisition/Designation  System  (MTADS)  Jitter 

•  Dual  Helmet  and  Display  Siting  (HADS)  Failure 

•  Remote  High  Frequency  (HF)  Safety  Fan 

•  Discrete  Selectable  Aircraft  Survivability  Equipment  (ASE)  Volumes 

•  Very  High  Frequency  (VHF)  Secure  Communications 

The  Enhanced  Transmission/Dual  Accessory  upgrade  was  also  chosen.  However, 
the  model  only  chose  to  purchase  358  of  these  upgrades,  instead  of  the  possible  948. 
Finally,  the  model  declined  to  purchase  the  UTA  Weight/Capability  and  the  Seat  Design 
upgrades.  With  a  budget  of  $12  million,  the  utility  model  calculated  the  optimum 
portfolio  depicted  in  Figure  12,  costing  $11,995,000. 

The  control  results  show  very  clearly  that  in  “optimization  mode”,  with  all  weight 
scores  set  at  the  default  value  of  1,  the  utility  model  is  choosing  upgrades  that  allow  the 
cheapest  addition  of  quality  to  the  AB3  fleet.  The  utility  model  chose  all  of  the  software- 
related  upgrades  (binary,  fleet-wide  purchases).  Additionally,  the  utility  model  chose 
fleet -wide  purchases  (all  790  helicopters  plus  20%  spares  =  948)  of  each  hardware 
upgrade  with  a  cost  per  quality  point  of  $39.24  and  below.  The  primary  reason  for  the 
partial  purchase  decision  (Dual  Accessory  Upgrade)  was  also  the  cost  per  quality  point. 
This  upgrade  had  a  cost  per  quality  point  of  $181.16,  indicating  a  price  sensitivity  zone 
wherein  the  model  begins  to  find  the  marginal  cost  per  quality  point  to  be  inequitable. 
Neither  the  UTA  Weight/Capability  nor  the  Seat  Design  upgrades  were  chosen  for 
purchase  due  to  costs  per  quality  point  of  $455.80  and  $366.27,  respectively.  These  “no 
buy”  decisions  provide  further  evidence  that  a  price  sensitivity  zone  exists,  as  mentioned 
previously.  These  results  prove  that  the  measurement  of  marginal  cost  is  central  to  the 
model’s  purchase  decisions. 
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Our  first  experiment  utilized  rank  weighting  in  the  utility  model  worksheet.  The 
weighting  scores  range  from  16  (highest  rank)  to  1  (lowest  rank).  The  assigned  scores 
were  randomized  using  Excel.  Figure  13  displays  the  input  data  for  the  first  experiment. 
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Figure  13.  First  Experiment:  Data 
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The  randomly  assigned  rank  scores  are  displayed  across  the  second  row  of 
Figure  13.  With  rank  scores  entered  into  the  worksheet,  the  QIS  scores  for  the  16 
potential  upgrades  change  dramatically,  ranging  from  69.6  to  986.1.  The  average  QIS 
score  was  349.9.  Six  of  the  potential  upgrades  scored  above  400  points,  and  the 
remaining  10  scored  between  69.6  and  367.2  points.  Regression  analysis  showed  that 
with  ranking  scores  in  place,  the  two  most  influential  factors  in  detennining  the  QIS  were 
the  Rank  and  Percent  Improved  Capability  factors.  The  results  of  the  first  experiment  are 
shown  in  Figure  14. 
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Figure  14.  First  Experiment:  Results 


41 


These  results  reflect  preferences  that  were  introduced  into  the  system  with  the 
randomized  ranking  scores.  The  utility  model  declined  to  purchase  three  of  the  eight 
software-related  potential  upgrades  (Decaying  Rotor  Indication,  FM  Muting,  and  TADS 
Failure  Weapon  Inhibit).  This  choice  is  in  contrast  with  the  control  results,  in  which  the 
utility  model  chose  to  purchase  all  eight.  Further,  in  the  presence  of  ranking  inputs,  the 
model  declined  to  purchase  the  Enhanced  Transmission/Dual  Accessory  or  Discrete, 
Selectable  ASE  Volumes  upgrades  in  significant  quantities.  Surprisingly,  the  utility 
model  chose  to  purchase  only  232  of  the  possible  948  Seat  Design  upgrades,  despite  that 
upgrade  being  ranked  second.  On  further  analysis,  we  attribute  this  decision  to  the  high 
cost  of  the  Seat  Design  upgrade  ($26,371  per  platfonn). 

The  utility  model  also  decided  to  purchase  a  significant  amount  (284)  of  the 
Remote  HF  Safety  Fan  upgrade,  despite  that  it  was  ranked  twelfth.  As  in  the  Seat  Design 
decision,  the  utility  model  used  the  relative  value  to  influence  the  purchase  decision  since 
the  HF  Safety  Fan  costs  $79  per  platfonn.  Statistical  analysis  showed  the  conelation 
coefficient  between  quantities  purchased  and  rank  to  be  0.77  (very  strong).  However,  the 
results  also  showed  that  rank  alone  was  not  a  strong  enough  factor  to  entirely  influence 
the  utility  model’s  decisions.  Cost  and  value  remained  relevant  factors.  The  correlation 
coefficient  between  cost-per-quality  point  and  quantity  purchased  was  -0.43,  showing  a 
significant  negative  relationship. 

The  results  of  this  experiment  show  that  the  ranking  method  can  be  used  as  an 
effective  way  to  apply  a  weighted  influence  in  a  multiple  criteria  decision-making 
(MCDM)  framework,  namely  this  utility  model.  They  also  show  that  the  ranking  method 
creates  an  appropriate  and  measured  influence  in  purchase  decisions,  as  seen  in  the  Seat 
Design  and  Remote  HF  Safety  Fan  upgrades.  In  summary,  a  high  rank  score  can  help  a 
potential  upgrade  in  the  utility  model’s  calculations,  but  cost  remains  an  important  factor. 

Our  second  experiment  utilized  the  summation  weighting  method.  This  method 

assigns  weighted  value  to  entities  according  to  an  ordinal,  pre-determ ined  priority 

(chronology,  cost,  size,  etc.).  For  the  purpose  of  this  experiment,  we  chose  a  non- 

influential  priority,  and  the  16  potential  upgrades  were  assigned  summation  weight  values 

according  to  alphabetic  order.  Upgrades  with  titles  starting  closer  to  the  beginning  of  the 
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alphabet  were  assigned  lower  scores,  and  those  with  names  starting  closer  to  the  end  of 
the  alphabet  received  higher  scores.  Table  3  displays  the  summation  score  assignment 
for  this  experiment  in  detail.  Figure  15  displays  the  data  for  the  second  experiment. 


Table  3.  Summation  Score  Assignment  for  the  Second  Experiment 


Upgrade  Name 

AH-64E  MTADS  Jitter 

Certified  PERF  Page 

CMWS  Indication 

Decaying  Rotor  Indication 

Discrete,  Selectable  ASE 

Volumes 

Dual  HADS  Failure 

Enhanced  Transmission  / 

Dual  Accessory 

FM  Muting 

Hydraulic  Pressure  Digital 

Readouts 

Opposite  Seat  Fixed  Gun 
Message 

Remote  HF  Safety  Fan 
(Display) 

Seat  Design 

Secure  Communications 

TADS  failure  Weapon 
Inhibit 

UTA  Weight/Capability 
(C,L,S  &  UHF) 

VHF  Secure 

Communications 

Summation  Weight 

1 

1 

2 

2 

3 

3 

4 

4 

5 

5 

6 

6 

7 

7 

8 

8 
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Figure  15.  Second  Experiment:  Data 
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The  alphabetically  assigned  summation  scores  are  displayed  across  the  second 
row  of  Figure  15.  With  summation  scores  entered  into  the  worksheet,  the  QIS  scores  for 
the  16  potential  upgrades  ranged  from  67.2  to  314.1.  The  average  QIS  score  was  194.8. 
Seven  of  the  potential  upgrades  scored  above  200  points,  and  the  remaining  nine  scored 
between  67.2  and  196.2  points.  Regression  analysis  showed  that  with  summation  scores 
in  place,  the  three  most  influential  factors  in  determining  the  QIS  were  the  Summation 
Score,  TRL  Level,  and  Percent  Improved  Capability  factors.  The  results  of  our  second 
experiment  are  shown  in  Figure  16. 
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Figure  16.  Second  Experiment:  Results 
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The  results  of  our  second  experiment  reflect  influences  that  were  introduced  into 
the  system  with  an  alphabetized  summation  score  assignment.  The  utility  model  declined 
to  purchase  only  one  of  the  eight  software-related  potential  upgrades  (Certified  PERF 
Page).  In  contrast  to  all  other  results  up  to  this  point,  the  utility  model  purchased  some  of 
each  hardware  upgrade.  In  the  control  results,  the  utility  model  chose  all  eight  software 
upgrades  and  six  of  the  eight  hardware  upgrades.  Despite  being  ranked  first,  the  utility 
model  only  chose  to  purchase  95  of  the  possible  948  UTA  Weight/Capability  upgrades. 
Investigation  showed  that  this  decision  was  due  to  the  high  cost  ($36,919  per  platform)  of 
UTA  Weight/Capability  upgrades.  The  utility  model  also  decided  to  purchase  the 
entirety  (948)  of  the  Discrete,  Selectable  ASE  Volume  upgrade,  despite  that  it  was  ranked 
seventh.  Like  the  UTA  Weight/Capability  upgrade,  the  utility  model  factored  value  into 
the  calculation  because  the  Discrete,  Selectable  ASE  Volume  upgrade  is  $1,265  per 
upgrade.  Statistical  analysis  showed  the  correlation  coefficient  between  quantities 
purchased  and  summation  rank  to  be  0.19  (weak).  The  outcome  of  this  experiment 
revealed  that  in  an  assessment  of  this  size,  summation  weighting  does  not  account  for  a 
significant  amount  of  the  purchase  decision.  Cost  per  upgrade  proved  to  be  the  most 
influential  factor.  The  correlation  coefficient  between  cost-per-quality  point  and  quantity 
purchased  was  -0.76,  showing  a  very  strong  negative  relationship. 

The  results  of  our  second  experiment  show  that  the  summation  ranking  method  is 
a  viable  option  for  weighted  utility  modeling  in  order  to  derive  satisficing  answers  within 
this  utility  model.  However,  this  method  should  only  be  used  in  an  assessment  of  ten 
upgrades  or  more  when  PMs  wish  to  make  a  minor  impact  on  the  calculations  of  the 
utility  model.  In  an  assessment  of  this  size,  the  summation  method  creates  a  much 
smaller  influence  than  the  ranking  method.  With  the  summation  technique  in  place,  a 
high  rank  score  is  not  enough  to  cause  a  fully  devoted  purchase  decision  within  the  utility 
model.  In  order  to  be  chosen  for  large  quantity  purchases,  an  upgrade  must  present  a 
good  value. 

Our  third  experiment  tested  the  use  of  priority  ranking  in  the  utility  model. 
Prioritization  uses  a  set  of  integer  values  to  assign  scores  based  on  relative  importance. 
The  two  most  common  applications  of  this  method  utilize  either  a  1  through  5  or  1 
through  3  ranking  system.  In  this  research  project,  we  utilized  the  1  through  3  value 
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system  (3  is  high  and  1  is  low).  Priority  ranks  were  randomly  assigned  to  the  potential 
upgrades  using  Excel.  Figure  17  shows  the  data  from  the  third  experiment. 
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Figure  17.  Third  Experiment:  Data 
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The  randomly  assigned  priority  scores  are  displayed  across  the  second  row  of 
Figure  17.  With  priority  scores  entered  into  the  worksheet,  the  QIS  scores  for  the  16 
potential  upgrades  ranged  from  66  to  233.4.  The  average  QIS  score  was  133.4.  Four  of 
the  potential  upgrades  scored  above  150  points,  and  the  remaining  1 1  scored  between  66 
and  139.2  points.  Regression  analysis  showed  that  with  priority  scores  in  place,  there  are 
four  influential  factors  in  determining  QIS:  Priority,  Percent  Improved  Capability, 
Contracting  Ease,  and  Reliability  (listed  in  order  of  influence).  The  results  of  our  third 
experiment  are  shown  in  Figure  18. 
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Figure  18.  Third  Experiment:  Results 
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The  results  of  the  third  experiment  reveal  the  impacts  of  randomized  priority 
values  within  the  utility  model.  Like  the  optimized  (control)  solution  set,  the  utility 
model  did  not  purchase  the  Seat  Design  or  UTA  Weight  Capability  upgrades  in  the 
presence  of  priority  values.  These  decisions  can  be  attributed  to  the  high  cost  of  the  two 
upgrades.  Contrary  to  the  control  results,  in  the  third  experiment,  the  model  did  not 
purchase  the  TADS  Failure  Weapon  Inhibit  software-related  upgrade,  despite  its 
relatively  low  cost  ($50  per  upgrade).  Analysis  showed  that  this  decision  was  made 
because  the  TADS  Failure  Weapon  Inhibit  upgrade  received  the  lowest  possible  priority 
score  (a  1  out  of  a  possible  3).  The  low  priority  score  contributed  significantly  to  a  low 
QIS  score  (a  66  in  this  instance).  Statistical  analysis  showed  the  correlation  coefficient 
between  quantities  purchased  and  priority  rank  to  be  0.78  (very  strong).  The  outcome  of 
this  experiment  revealed  that  in  an  MCDM  assessment,  priority  ranking  makes  a 
meaningful  influence  on  the  purchase  decision.  Cost  per  upgrade  proved  to  be  the  most 
influential  factor.  The  correlation  coefficient  between  cost-per-quality  point  and  quantity 
purchased  was  -0.76,  showing  a  very  strong  negative  relationship. 

The  results  from  the  third  experiment  show  that  prioritization  can  be  used  in 
MCDM.  This  method  produced  results  that  resemble  the  control  results  very  closely. 
For  this  reason,  we  conclude  in  this  research  project  that  use  of  prioritization  in  an 
assessment  of  this  size  should  be  used  only  when  PMs  wish  to  make  a  small,  but 
meaningful  impact  on  the  results  of  the  utility  model’s  assessment.  This  method 
noticeably  influences  purchase  decisions  for  upgrades  that  receive  the  highest  (and 
lowest)  priority.  Upgrades  that  are  left  in  the  middle  are  not  given  enough  influence  and 
are  chosen  based  on  their  value  alone.  Upgrades  that  are  given  the  highest  scores  are 
always  chosen,  unless  they  are  prohibitively  expensive,  and  those  with  the  lowest  scores 
are  not  chosen  unless  they  provide  high  quality  at  a  low  cost. 

We  designed  these  experiments  to  test  how  the  different  methods  affect  the  results 
of  the  utility  model  produced  in  this  research  project.  They  also  served  to  ensure  that  the 
utility  model  would  indeed  factor  the  influences  of  weighted  utility  modeling  into  its 
calculations.  The  outcomes  of  the  experiments  allowed  us  to  better  understand  how  each 
method  pairs  with  different  MCDM  scenarios.  These  methods  have  varying  impacts  on 
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the  results  of  the  utility  model’s  calculations.  As  discussed  in  the  results  of  each 
experiment,  some  methods’  impacts  are  less  subtle  and  should  only  be  used  by  PMs  when 
appropriate.  Table  4  provides  a  cross-section  of  the  experimental  results  and  the 
correlative  relationships  between  the  different  weighted  utility  modeling  methods  and 
quantity  purchase  decisions. 


Table  4.  Experiment  Results:  Correlations  to  Quantities  Purchased 


Correlation  to  Quantity  Purchased 

Utility  Model  Type 

Factor 

Optimization 

Weig 

ited  Utility  Modeling 

Ranking 

Summation 

Prioritization 

QIS 

-0.112212194 

0.745339082 

0.151922701 

0.559090478 

Price/Upgrade 

-0.930849728 

-0.367408395 

-0.732071087 

-0.733143384 

Input 

N/A 

0.769445798 

-0.01687123 

0.772208714 
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V.  UTILITY  MODEL  APPLICATIONS  FOR  PROGRAM 

MANAGEMENT 

A.  ASSESSMENT  OF  THE  MODEL’S  OVERALL  UTILITY 

The  AB3  Modernization  PM  populated  the  worksheet  of  the  utility  model  with  the 
quality  data  from  16  potential  upgrades.  In  this  research  project,  the  authors  used  this 
data  to  employ  the  model  and  generate  an  optimum  solution.  After  the  AB3 
Modernization  PM  was  presented  with  the  results  displayed  in  Figure  12,  he  felt  the 
model  would  be  a  useful  tool  for  educating  his  staff  about  the  implications  of  their 
decisions  in  the  form  of  procurement  trade-offs.  With  regard  to  overall  utility,  the  PM 
gave  the  model  a  rating  of  8  out  of  10.  The  PM  also  felt  such  a  utility  model  would  be 
very  useful  for  new  PMs  because  it  would  help  them  to  quickly  become  familiar  with  the 
price  and  quantity  sensitivities  in  their  respective  programs.  For  example,  an  increase  in 
the  quantity  purchased  of  a  certain  item  implies  a  decrease  for  another.  The  utility  model 
also  proved  useful  in  analyzing  the  impacts  that  price  increases  or  decreases  had  on 
quantities  of  upgrades  that  could  be  purchased  and  the  second  and  third  order  effects  to 
the  program’s  overall  optimal  value.  Finally,  the  PM  stated  that  the  model  could  be 
useful  for  conducting  what-if  scenarios,  such  as  examining  the  impacts  of  budget 
cuts/increases  and  reporting  these  impacts  to  higher  levels  of  decision-making  authority. 

B.  CREATION  OF  THE  PROGRAM  DASHBOARD 

One  the  benefits  of  consolidating  information  such  as  pricing,  budget  constraints, 
and  quality  factors  into  one  location  is  the  creation  of  a  “dashboard”  style  method  of 
presenting  the  current  status  of  the  MDAP  block  upgrade  program.  A  dashboard  is  the 
consolidated  presentation  of  information  that  has  been  pulled  from  various  sources 
throughout  an  organization  in  a  form  that  is  easy  to  interpret.  With  all  relevant  data 
consolidated,  it  is  much  easier  for  a  decision-maker  to  connect  all  the  dots  and  to  make 
better  decisions  when  the  impacts  of  a  decision  can  be  seen  throughout  the  program 
rather  than  in  a  singular  context.  Another  useful  function  of  a  dashboard  is  the  ability  to 
get  a  current  assessment  or  situational  report  (SITREP)  of  where  a  program  or  mission 
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currently  stands.  This  is  very  valuable  to  decision-makers  because  they  can  rapidly 
respond  to  unforeseen  events  rather  than  take  additional  time  to  understand  the  current 
situation  and  then  make  decisions. 

The  utility  model  can  provide  a  dashboard  for  the  PM  of  any  program  if  properly 
customized.  By  pulling  the  relevant  cost,  quality,  and  other  constraints  into  a  single 
interface,  PMs  can  always  have  visibility  as  to  where  their  program  currently  stands.  As 
any  one  category  changes  or  updates  (for  example,  a  contractor  provides  a  final  price  for 
a  potential  upgrade),  the  update  is  immediately  made  to  the  utility  model  and  the  overall 
program  can  be  recalculated.  By  having  this  granular  level  of  instant  situational 
awareness,  decision-making  is  simplified  and  improved  for  the  overall  benefit  of  the 
program. 

C.  SENSITIVITY  ANALYSIS 

In  their  book,  Sensitivity  Analysis,  Saltelli,  Chan,  and  Scott  (2000)  defined 
sensitivity  analysis  as  “the  study  of  how  variation  in  the  output  of  a  model  can  be 
apportioned,  qualitatively  or  quantitatively,  to  different  sources  of  variation,  and  of  how 
the  given  model  depends  upon  the  infonnation  fed  into  it”  (p.3).  Our  research  project 
found  that  factor  screening  and  global  sensitivity  analysis  were  most  practical  for  this 
utility  model.  In  order  to  provide  the  most  useful  infonnation  to  PMs,  the  factor 
screening  methods  were  limited  to  one  at  a  time  (OAT)  global  sensitivity  analysis.  OAT 
analysis  calls  for  the  manipulation  of  one  variable,  factor,  or  constraint  at  a  time  in  order 
to  assess  the  impact  to  the  model’s  outcome.  This  sort  of  sensitivity  analysis  is  the  most 
useful  for  the  purposes  of  this  research  project  because  it  enables  PMs  to  detennine  the 
relationships  between  individual  quality  factors,  constraints,  and  QIS  scores  and  their 
relationships  to  the  optimum  solution.  Infonnation  about  these  relationships  can  help 
PMs  better  understand  which  factors  are  more  important  in  adding  quality  to  a  platfonn 
and  which  constraints  are  the  most  limiting. 

By  using  our  utility  model,  PMs  can  pose  hypothetical  questions  and  assess 
potential  program  impacts.  In  this  research  project,  we  tested  the  model’s  sensitivity 
analysis  capability  with  the  evaluation  of  potential  upgrades  from  the  AB3  Modernization 
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Program.  During  the  application  experiment  of  our  model,  the  AB3  Modernization  PM 
changed  the  AB3  Modernization  Program  flexible  budget  amount,  upgrade  quantities, 
and  upgrade  prices.  As  a  result,  the  PM  was  able  to  assess  many  possible  scenarios 
simply  by  changing  a  few  inputs  within  the  worksheet  portion  of  the  utility  model.  In 
one  sensitivity  analysis  assessment,  the  PM  reduced  the  AB3  Modernization  Program’s 
flexible  budget  by  25%.  As  a  result,  the  PM  was  able  to  instantly  see  the  reduction  in 
upgrade  quantities  the  program  was  able  to  buy.  Additionally,  the  PM  was  able  to  assess 
new  price  points  for  upgrades  that  would  allow  the  program  to  purchase  them  in  the  same 
quantities.  The  PM  “asked”  the  model  all  of  these  questions  and  received  quantitative 
answers  within  a  matter  of  minutes.  The  implication  is  that  PMs  could  use  sensitivity 
analysis  to  simulate  or  “war-game”  any  conceivable  scenario.  The  AB3  Modernization 
PM  noted  that  this  analysis  capability  is  very  valuable  because  it  would  allow  him  to 
simulate  changes  to  the  most  important  factors  that  affect  his  program  overall. 

Sensitivity  analyses  regarding  budget  and  prices  are  arguably  the  most  valuable  to 
PMs.  However,  the  flexible  nature  of  the  utility  model,  and  the  customization  of  tune  as 
a  quality  factor  (specifically  for  the  AB3  program),  would  allow  PMs  to  conduct  analyses 
on  any  combination  of  quality  factors.  Because  the  model  “sees”  all  of  these  factors  as 
variables  in  an  equation,  they  can  be  changed  ad  hoc,  and  new  results  can  be  produced  for 
comparison.  Results  of  sensitivity  analyses  regarding  any  of  the  quality  factors  that  have 
been  programmed  in  for  the  AB3  Modernization  Program  could  potentially  ann  the  AB3 
Modernization  PM  with  quantitative  data  when  negotiating  with  vendors,  fellow  PMs,  or 
PEO  Aviation  for  more  money,  lower  prices,  better  quality,  and  so  forth. 

Another  vital  purpose  of  sensitivity  analysis  is  to  test  the  accuracy  of  a  model.  In 
fact,  Saltelli  et  al.  (2000)  stated  that  a  mathematical  model  is  not  truly  complete  without  a 
built-in  capability  to  assess  its  accuracy.  The  utility  model  we  produced  in  this  research 
project  is  no  exception  to  that  rule.  In  order  to  produce  this  capability,  we  added  a 
portion  to  the  model;  an  additional  tab  in  the  Excel  workbook  titled  Sensitivity  Analysis. 
This  tab  is  a  carbon  copy  of  the  Solver  tab,  except  that  the  decision  variables  were  not 
programmed  as  integers.  This  allowed  Microsoft  Excel’s  Solver  add-in  to  produce  an 
itemized  sensitivity  analysis  report.  PMs  could  use  this  report  to  break  out  information 
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regarding  shadow  prices,  allowable  price  increases,  and  decreases  for  each  variable  of  the 
model.  This  feature  would  provide  PMs  with  a  snapshot  of  sensitivities.  We  used  this 
same  report  in  this  research  project  to  assess  the  accuracy  of  the  utility  model. 

D.  WEIGHTED  UTILITY  MODELING 

In  this  research  project  we  approached  the  MDAP  block  upgrade  process  as  an 
instance  of  multiple  criteria  decision-making  (MCDM).  The  primary  goal  of  the  utility 
model  we  produced  in  this  research  project  is  optimization  within  the  MCDM 
framework.  That  is,  this  utility  model  first  seeks  to  find  the  best  possible  solution  by 
working  with  multiple  decision  variables  and  constraints.  This  objective  assumes  no 
appointed  motivations  on  the  part  of  the  PM.  However,  there  are  many  instances  in 
MDAP  block  upgrade  programs  when  the  PM  does  have  predetermined  goals  and 
priorities.  These  arise  from  operational  need,  budgetary  constraints,  or  any  other  number 
of  exterior  influences.  In  examples  where  the  PM  has  clear  goals  to  achieve  within  the 
MDAP  upgrade  program,  weighted  utility  modeling  comes  into  play. 

If  the  AB3  Modernization  PM  were  given  a  directive  to  achieve — such  as  give  the 
AB3  Modernization  Program  25%  greater  weight  capacity  and  make  the  seat  more 
ergonomic-then  he  would  have  clear  goals  to  work  toward.  In  the  presence  of  these 
weighted  variables,  all  non-related  upgrades  would  take  a  backseat  to  those  that  would 
allow  the  PM  to  accomplish  the  goal  of  adding  weight  capacity  and  making  the  seat  more 
ergonomic.  Fortunately,  the  PM  can  program  goals  like  this  into  the  utility  model.  The 
weight  factor  in  the  Excel  worksheet  of  the  utility  model  allows  the  PM  to  use 
lexicographic  (ordering  or  ranking),  or  weighted  utility  modeling  methods,  in  order  to 
accommodate  any  predetermined  goals  into  calculations.  Use  of  these  methods 
essentially  changes  the  objective  of  the  model  from  optimization  to  satisficing,  or  finding 
the  answer  that  the  decision-maker  needs. 

When  this  utility  model  was  applied  to  the  AB3  Modernization  Program,  there 
were  no  programmatic  goals.  The  AB3  Modernization  PM  simply  wanted  to  know  what 
the  best  mix  of  upgrades  to  purchase  would  be.  In  other  words,  he  was  looking  for  an 
optimized  solution.  However,  in  future  scenarios,  PMs  could  certainly  apply  weighted 
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utility  modeling  in  order  to  find  solution  sets  that  satisfice  their  goal  requirements.  In 
order  to  simulate  the  effects  of  weighted  utility  modeling  in  producing  the  results  that 
PMs  need,  we  conducted  a  simulation  that  reflects  the  scenario  described  previously.  The 
PM  is  given  the  directive  to  add  weight  capacity  and  a  more  ergonomic  seat.  Figures  19 
through  22  present  the  optimized  and  satisficing  inputs  and  results  from  this  simulation. 
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Figure  19.  Optimization  Data  Input 
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Figure  20.  Optimization  Results 
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Figure  2 1 .  Weighted  Utility  Modeling  Data  Input 
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Figure  22.  Weighted  Utility  Modeling  Results 
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The  results  displayed  in  Figures  19  through  22  clearly  show  that  weighted  utility 
modeling  can  be  used  to  find  satisficing  solutions  in  the  presence  of  predetennined  goals. 
In  the  control  results,  before  weighted  utility  modeling  is  applied,  neither  the  UTA 
Weight  nor  the  Seat  Design  upgrades  are  selected  by  the  utility  model.  However,  once 
weighting  utility  modeling  is  applied,  the  utility  model  finds  a  solution  that  satisfices  the 
goals  while  still  selecting  other  upgrades  in  quantities  that  add  quality  to  the  portfolio. 

Additionally,  there  may  be  instances  in  an  MDAP  block  upgrade  program  that 
require  constraints  of  mutual  exclusivity.  These  constraints  stipulate  tenns  such  as 
“upgrade  X  can  only  be  purchased  if  upgrade  Y  is/is  not  purchased.”  Another  unique 
quantity  constraint  is  the  k  of  n  constraint,  which  indicates  that  “of  upgrades  X,  Y,  and  Z, 
only  two  can  be  purchased.”  This  research  project  addresses  these  types  of  unique 
constraints  in  detail  in  Chapter  III.  In  events  where  these  constraints  are  present, 
weighted  utility  modeling  could  be  applied  to  the  variables  one  at  time  to  counter  the 
mathematical  deviations  encountered  within  Excel  Solver  in  order  to  derive  a  solution. 
Solutions  of  this  nature  may  not  assist  PMs  in  quantity-based  decision-making. 
However,  these  solution  sets  could  assist  PMs  in  developing  a  better  understanding  of 
finite  values  within  the  upgrade  program  and  their  relationships  to  one  another. 

In  other  MDAP  block  upgrade  programs,  PMs  may  bear  a  heavy  disposition 
toward  certain  upgrades  due  to  exterior  influences  such  as  DoD  objective  programming 
or  public/Congressional  pressure.  Weighted  utility  modeling  has  significant  value  when 
PMs  need  the  model  to  “choose”  a  pre-detennined  upgrade  in  a  set  quantity  and  still 
reach  an  optimal  solution  that  includes  all  potential  upgrades. 

E.  EDUCATION  OF  PROGRAM  STAFF 

Ensuring  that  each  member  of  a  program  staff  has  good  situational  awareness  of 
the  program  will  reduce  misunderstandings  and  miscommunications  both  internally  and 
externally.  Lapses  in  situational  awareness  often  lead  to  friction  within  a  program  that 
could  result  in  unnecessary  delays  and  costs  and  in  less  than  optimal  value  for  the 
taxpayer.  By  using  the  utility  model  in  a  dashboard  format  for  all  of  the  relevant  staff  in 
a  program,  the  staff  becomes  educated  about  the  effects  their  individual  decisions  have  on 
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the  overall  program.  This  helps  each  member  to  see  the  overall  picture  and  to  make  more 
informed  decisions  within  their  scope  of  responsibility.  The  AB3  Modernization  PM  felt 
this  type  of  education  would  help  his  staff  meet  the  overall  program  goals.  Additionally, 
this  model  would  help  newer  staff  members  leam  the  breadth  of  the  program  by  seeing 
how  all  the  program’s  parts  interact  with  each  other. 

F.  DECISION-MAKING  SUPPORT 

In  large  programs  where  dozens  of  stakeholders  are  involved  and  budgets  are  in 
the  hundreds  of  millions  of  dollars,  it  is  often  difficult  to  make  a  good  and  complete 
decision  because  it  is  hard  to  see  decision  alternatives  and  because  not  all  necessary 
information  is  available.  When  combining  uncertainty  such  as  future  budget  amounts, 
requirements  changes,  quality  factors,  unforeseeable  issues,  and  large  dollar  amounts,  it 
is  vital  that  each  decision  be  thoroughly  researched  since  poor  decisions  can  be  very 
disruptive  and  expensive.  The  use  of  our  utility  model  could  reduce  some  of  the 
unknown  implications  a  decision  carries  and  could  result  in  better  overall  decisions. 
Decision-makers  must  first  define  the  problem  (ID  constraints  and  quality  factors),  list 
the  alternatives  (customize  the  worksheet),  identify  future  outcomes  for  each  alternative 
(sensitivity  analysis),  identify  payoffs  or  costs  for  each  decision  (run  the  utility  model), 
and  finally  make  a  decision.  The  utility  model’s  results  feed  nicely  into  other  decision¬ 
making  models  such  as  maxi-max/maxi-min,  criterion  of  realism,  equally  likely,  or 
decision  trees.  By  combining  the  utility  model  and  other  decision-making  models  as  a 
compound  data  analysis  tool,  PMs  can  make  more  accurate  and  robust  decisions 
(Balakrishnan  et  al.,  201 1). 

The  model  can  provide  quantitative  support  for  recommendations  a  PM  might 
make  above  his  own  echelon  (such  as  at  the  PEO  level).  The  utility  model  provides  the 
ability  to  conduct  various  decision  analysis  techniques,  which  in  the  long  run  will  result 
in  successful  outcomes  for  the  entire  organization  if  conducted  properly. 

G.  PROGRAM  MANAGEMENT  OFFICE  CONTINUITY 

Continuity  may  be  the  lifeblood  of  a  program  management  office.  Over  the 

course  of  an  MDAP’s  life,  it  is  certain  that  there  will  be  multiple  changes  to  all  of  the 
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factors  that  directly  affect  its  success.  Because  the  DoD  uses  evolutionary  acquisition 
(EA)  as  its  preferred  method  for  the  acquisition  of  platfonns  such  as  the  AH-64  Apache 
attack  helicopter,  the  program/project  management  office  (PMO)  could  last  decades. 
Over  those  years,  the  nature  of  the  program,  its  budgetary  constraints,  its  systems  and 
personnel  are  subject  to  significant  turnover.  No  researcher  could  hope  to  develop  a 
utility  model  that  would  encompass,  process,  and  manage  all  of  that  change.  However,  in 
the  right  hands,  a  utility  model  can  help  PMs  navigate  change. 

MDAPs  are  subject  to  change  with  the  needs  and  requirements  of  the  DoD. 
Therefore,  a  program  that  is  alive  and  healthy  one  day  may  be  cancelled  the  next  due  to 
budgetary  constraints.  This  utility  model  is  designed  to  help  PMs  optimize  a  solution  set 
in  an  MCDM  framework.  If  the  AB3  Modernization  Program  were  declared  to  be  the 
final  installment  of  the  Apache  block  upgrades,  then  the  PM  would  have  to  make 
decisions  about  how  to  close  out  the  program  in  the  most  economic  manner.  This  utility 
model  could  easily  be  customized  with  decision  variables  that  would  assist  in  this 
process. 

PMs  understand  that  nothing  can  be  accomplished  without  budgetary  resources, 
and  the  more  a  PMO  has,  the  more  it  can  accomplish  for  the  MDAP.  However,  money  is 
never  guaranteed,  and  it  often  varies  greatly  from  year  to  year.  During  tunes  of  conflict, 
MDAPs  often  enjoy  a  greater  amount  of  funding  in  order  to  add  capability  for  warfighter 
use.  In  tunes  of  peace,  money  is  often  programmed  away  from  the  DoD  and  thus  the 
wells  for  the  MDAPs  dry  up.  PMOs  are  forced  to  do  more  with  less.  Although 
maintaining  continuity  in  these  tunes  can  be  difficult,  a  utility  model  can  help  PMs 
maintain  funding  for  priority  efforts.  It  can  even  help  PMs  prepare  and  simulate  future 
fiscal  constraints  if  budget  cuts  are  looming.  Performing  simulations  can  help  PMs  to 
prepare  future  courses  of  action  (COAs)  for  their  successors. 

As  mentioned  previously,  there  are  tunes  when  the  role  of  an  MDAP  may  be 

expanded.  These  expansions  can  result  from  another  program’s  cancellation,  from  a 

contemporary  requirement,  or  from  new  developments  in  technology.  An  example  of 

role  expansion  is  the  AB3  Modernization  Program’s  new  role  to  simultaneously  manage 

and  utilize  unmanned  aerial  systems  (UAS)  while  on  station.  With  each  new  role  taken 
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on  by  a  platfonn  comes  a  suite  of  new  systems.  In  order  to  accommodate  the  new  UAS- 
related  role,  the  AB3  Modernization  Program  added  the  Manned/Un-manned  Teaming 
(MUT)  suite  of  systems.  The  addition  of  a  new  system  to  a  platfonn  can  prove 
complicated.  Adding  five  systems  simultaneously  could  closely  represent  chaos. 
However,  use  of  a  utility  model  implies  a  simple  adaptation  of  the  model  to 
accommodate  the  new  systems.  Simulations  can  immediately  be  run  to  detennine  what 
impact  the  presence  of  the  new  systems  will  have  on  budget  and  purchase  quantities. 

Personnel  are  also  subject  to  change  within  a  PMO.  Military  PMs  often  serve  in 
their  billet  for  three  years  or  less,  while  their  civilian  counterparts  remain  in  their 
positions  for  an  average  of  12  years  (Riley  &  Fallesen,  2013).  With  this  much  turnover 
among  military  leaders,  it  is  easy  to  imagine  how  data  could  get  lost  in  the  shuffle. 
However,  if  data  corresponding  to  cost,  quality,  and  schedule  are  stored  and  maintained 
using  a  utility  model,  then  there  is  less  chance  that  vital  information  will  become 
outdated  or  slip  through  the  cracks  as  leaders  transition  in  and  out  of  their  roles  within  a 
PMO. 

In  conclusion,  there  is  no  single  tool  that  can  serve  as  a  “catch  all”  to  maintain 
100%  continuity  within  a  PMO.  A  utility  model  cannot  solve  all  continuity-related 
problems  that  arise  from  changes  in  a  program’s  situation,  budget,  systems,  or  personnel. 
Instead,  a  utility  model  represents  one  piece  of  a  potential  solution  set  that  can  assist  PMs 
in  managing  the  ever-changing  environment  that  defines  an  MDAP. 
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VI.  SUMMARY,  CONCLUSION,  RECOMMENDATIONS  FOR 

FUTURE  RESEARCH 


A.  SUMMARY 

Throughout  this  project,  we  have  attempted  to  answer  our  original  research 
question:  Can  utility  modeling  be  used  to  effectively  find  an  optimal  allocation  of 
upgrade  purchases  when  choosing  from  a  range  of  potential  upgrades? 

Using  the  AB3  Modernization  Program,  we  were  able  to  answer  the  research 
question  by  using  a  real-world  MDAP  upgrade.  The  AB3  Modernization  Program 
proved  to  be  an  excellent  illustration  to  answer  the  research  question  because  of  the  wide 
range  of  potential  upgrades  that  the  PM  was  considering  for  purchase.  At  the  time  this 
study  was  conducted,  the  AB3  Modernization  PM  was  considering  16  technologies  with 
different  functions,  prices,  schedule  impacts,  and  trade-offs.  These  upgrades  involved 
hardware  and  software  and  improved  the  AB3  platfonn  in  different  manners  and  degrees. 
The  challenge  for  the  PM  was  finding  a  way  to  optimally  distribute  his  budgetary 
resources  to  achieve  the  optimal  value  for  the  taxpayer  and  the  warfighter. 

To  effectively  employ  linear  programming,  the  objective  must  be  to  maximize  or 
minimize  the  objective  variable.  In  the  case  of  the  AB3  Modernization  Program,  the  goal 
was  to  maximize  taxpayer  value  and  provide  the  most  capable  platfonn  for  the  end  user. 
With  the  goal  of  maximizing  total  utility,  the  objective  function  is  to  maximize  the 
overall  quality  of  the  various  upgrades  being  considered  within  the  constraints  presented 
by  each.  To  assess  quality,  we  developed  a  method  to  detennine  the  marginal 
improvement  offered  by  each  potential  upgrade.  Because  each  potential  upgrade 
perfonned  a  different  function,  we  had  to  ensure  we  were  comparing  each  item  equally. 
To  do  this,  we  developed  an  Excel  worksheet  that  summed  the  various  quality  factors 
into  a  final  score.  This  final  score  was  called  the  quality  index  score,  or  QIS.  The  QIS  is 
the  value  to  be  maximized,  and  it  facilitated  the  use  of  linear  programming  in  our 
research. 
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The  goal  of  this  research  project  was  to  determine  whether  utility  modeling  could 
be  used  in  the  DoD  acquisition  decision-making  process.  In  order  to  answer  this 
question,  we  designed  and  programmed  a  utility  model  that  could  be  customized  to  fit 
any  DoD  MDAP  block  upgrade  program.  The  assessment  data  we  used  allowed  us  to 
answer  two  questions  that  were  central  to  our  thesis:  (1)  Does  the  utility  model  return  a 
feasible  solution?  and  (2)  What  are  the  results  of  the  PM’s  assessments? 

To  be  considered  functional,  the  utility  model  must  provide  purchase  decisions  for 
the  potential  upgrades  being  considered  while  maximizing  quality.  The  model  must 
provide  these  answers  in  two  modes:  optimizing  and  satisficing  or  weighted  utility 
modeling.  Finally,  the  model  must  provide  sensitivity  analysis  for  the  PM.  In 
optimization  mode,  there  is  no  way  for  a  PM  to  provide  subjectivity.  Regression  and 
correlation  analyses  of  the  results  from  this  mode  showed  that  the  utility  model  made 
purchase  decisions  based  purely  on  cost  and  QIS.  The  results  were  optimal  because  there 
was  no  other  possible  solution  that  offered  more  quality  for  the  AB3  fleet  within  the  same 
budget.  This  is  exactly  the  answer  that  the  optimization  mode  was  designed  to  find.  In 
weighted  utility  modeling  mode,  the  model  must  provide  the  PM  with  answers  that 
satisficed  certain  external  influences  or  tastes.  As  part  of  this  research  project,  we 
conducted  experiments  to  test  the  three  weighted  utility  modeling  methods  that  are  most 
relevant  to  MCDM:  ranking,  prioritization,  and  summation.  Analysis  of  the  results 
revealed  that  all  three  methods  are  relevant,  but  they  have  different  effects  on  the 
outcome  of  the  utility  model’s  decisions.  Further  analysis  helped  us  to  map  these  methods 
with  appropriate  scenarios.  In  both  modes  of  the  utility  model,  sensitivity  analysis  allows 
the  PM  to  determine  shadow  pricing  and  price/quantity  sensitivities.  Additionally, 
sensitivity  analysis  allowed  us  to  determine  the  accuracy  of  the  model’s  calculations. 

Analysis  of  the  model’s  results  allowed  for  better  understanding  of  the  logic  that 
drove  purchase  decisions.  In  this  research  project,  we  interpreted  the  purchase  decision 
results  in  tenns  of  the  quantity  to  purchase  metric.  We  developed  two  metrics  within  the 
utility  model  to  help  understand  and  track  our  results:  the  QIS  and  the  price-per-quality 
point.  Correlation  data  for  these  metrics  showed  that  price/value  is  always  an  overarching 
factor  in  decisions  (as  it  should  be).  Regression  analysis  also  showed  that  factors  such  as 
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percent  improved  capability  and  TRL  were  more  influential  than  others.  This  is 
appropriate  to  the  acquisition  decision-making  environment  because  technology  maturity 
and  marginal  benefits  often  drive  an  upgrade’s  success.  Data  analysis  allowed  for  a 
deeper  understanding  of  which  quality  factors  drive  overall  upgrade  attractiveness  and 
also  showed  how  relationships  with  cost  drive  optimized  and  satisficed  decisions. 

B.  CONCLUSION 

The  results  of  this  research  project  prove  that  linear  programming  and  utility 
modeling  can  positively  contribute  to  the  DoD  acquisition  decision-making  process  in 
several  ways.  A  PM,  or  a  member  of  the  program/project  management  team,  can  use 
utility  modeling  to  arrive  at  optimized  or  weighted  results  regarding  a  side-by-side 
comparison  of  many  potential  upgrades.  Further,  the  user  can  apply  a  sensitivity  analysis 
to  the  results  to  determine  important  factors  such  as  shadow  prices  and  price-to-quantity 
sensitivities. 

The  results  or  purchase  decisions  of  an  adequately  programmed  utility  model  are 
granular,  value  based,  and  easy  to  interpret.  Because  of  this,  they  have  many  uses  in  the 
acquisition  decision-making  environment.  PMs  can  apply  utility  modeling  at  the  outset 
of  a  program’s  phase  and  use  the  results  as  a  beginning  point  for  deeper  upgrade 
assessments  like  the  POM  process.  The  results  can  also  be  used  to  quantitatively 
demonstrate  a  PM’s  position  when  negotiating  with  the  PEO  for  more  funds/time  or  with 
a  vendor  for  better  prices/greater  quantity.  Further,  utility  modeling  can  be  used  to 
educate  staff  members  or  to  help  with  continuity  during  tunes  of  transition.  Finally, 
because  utility  modeling  is  flexible  and  provides  answers  quickly,  PMs  can  use  it  to 
simulate  any  situation  that  can  be  quantitatively  modeled.  In  this  capacity,  utility 
modeling  can  be  an  invaluable  planning  and  situational  analysis  tool. 

Utility  modeling  fills  a  very  important  void  in  the  acquisition  decision-making 
environment:  the  ability  to  quantitatively  and  simultaneously  compare  many  potential 
upgrades.  Utility  modeling  can  provide  PMs  with  an  unprecedented  level  of  situational 
awareness  and  understanding  within  their  program.  Nested  within  the  first  tenet  of 
USD(AT&L)  Kendall’s  (2012)  Better  Buying  Power  2.0,  utility  modeling  provides  PMs 
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with  the  ability  to  make  optimal  cost  trade-off  decisions  and  to  maximize  the  value  and 
quality  of  their  portfolios.  PMs  can  also  profit  from  mathematical  data,  plan  for  the 
future,  and  maintain  education  and  stability  within  the  PMO. 

C.  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

While  conducting  research  for  this  project,  we  identified  several  areas  in  which 
future  research  could  expand  or  improve  the  functionality  of  utility  modeling  as  it  is 
applied  to  program  management  within  the  DoD: 

•  Forecasting  involves  techniques  that  attempt  to  reduce  uncertainty.  A 
quality  forecast  can  help  PMs  make  a  good  prediction  about  what  will 
occur  in  the  future  (Hillier  &  Liebennan,  1995).  This  allows  PMs  to  make 
better  decisions  regarding  the  future  course  of  their  programs.  In  making 
better  predictions  about  future  issues,  PMs  must  understand  which 
program  data  is  paramount  and  then  map  it  to  the  proper  forecasting 
method. 

•  According  to  Hillier  and  Liebennan  (1995),  “Game  Theory  is 
mathematical  theory  that  deals  with  the  general  features  of  competitive 
situations  like  these  in  a  fonnal,  abstract  way.  It  places  particular 
emphasis  on  decision  making  processes  of  the  adversaries”  (p.470). 
Within  game  theory,  utility  modeling  can  be  applied  to  solve  a  game  with 
mixed  strategies.  How  would  PMs  utilize  game  theory  to  gain  an 
advantage  over  a  contractor  in  negotiating  prices,  quality  features  of 
upgrades,  or  other  negotiated  aspects  of  program  management?  Could 
forecasting,  simulations,  and  sensitivity  analyses  lead  to  predictions  that 
could  be  applied  to  utility  modeling  game  theory? 

•  Retroactive  Program  Analysis  (RPA)  is  the  application  of  utility  modeling 
to  past  MDAP  block  upgrade  programs.  It  is  applied  in  order  to  leam 
from  previous  successes  or  failures.  This  sort  of  analysis  could  benefit  a 
PMO  because  the  results  could  lead  to  a  deeper  understanding  of  why  past 
programs  were  effective  or  fruitless.  These  lessons  could  help  recreate 
successes  and  avoid  the  pitfalls  of  previous  programs.  PMOs  should  apply 
utility  modeling  to  past  MDAPs  to  gain  further  awareness  of  cost  trade¬ 
offs  and  the  positive  application  of  utility  modeling  in  MCDM.  RPA  is 
best  utilized  at  the  outset  of  a  DoD  acquisition  effort  or  in  an  academic 
setting.  Utility  modeling  could  be  applied  to  previous  DoD  acquisition 
programs  in  the  same  way  that  it  was  applied  to  the  AB3  Modernization 
Program  in  this  thesis. 
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